Linux: UDP buffer monitoring utility (udpstat)

Udp statistics tool

The goals

The goal for this utility was to:

1. Present /proc/net/udp in human readable form:
- timestamp
- local address - from hex to dot notated (in case of IPv6 in hex) + port
- remote address - from hex to dot notated (in case of IPv6 in hex) + port
- rx, tx queue - in bytes, kbytes, mbytes (constrolled by parameter)
- drops

2. Constant monitoring with configurable interval polling:
- statistics presented every n seconds
- monitor interval controlled by command line parameter
- number of polls controlled by command line interface

3. Print Per pid statistics - present udp statistics per pid (/proc//net/udp)

4. Print kernel configuration parameters: max size + utilization of the buffer (rx queue / rx max size & tx queue / tx max size)

Where it can be downloaded from?

It has become a part of the yapcg project. The latest version available here: http://code.google.com/p/yapcg/source/browse/measurements/perftools/udpstat should meet all the goals . You can give it a try - what you will need is ruby 1.9.3 with optparse, ostruct gems (should be available in standard installation, at least these was the case on my Ubuntu 12.10 box)

The syntax of the utility is:


$ udpstat
Usage: udpstat [options]

Specific options:
    -6, --ipv6                       Enable IPv6 stat parsing
    -K, --kilobytes                  Present values in kilobytes
    -M, --megabytes                  Present values in megabytes
    -p, --ppid pid                   PID of the process to be monitored

Common options:
    -h, --help                       Show this message
    -v, --version                    Show version

    - interval in which statistics are to be reported
       - the number of statistics to be shown

@2013 Krystian Brachmanski, Wroclaw

As previously mentioned it has been verified on Ubuntu 12.10 64-bit OS, running 3.5.0-21-generic kernel and ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux].

How to use it?

I always say that I learn the most on examples therefore I recommend to check another article on my blog to see how you can truoubleshoot a simulated udp buffer problem in the application.

Ubuntu 12.10: UDP buffer monitoring (kernel 3.5.0-21)

Some time ago I was involved in analysis of a networking problem, the symptom of which were client-side timeouts (no answer received). The protocol was standardized, based on UDP.

Let's first analyze the flow of information from network adapter towards the application. There is an excellent document provided by RedHat as part of their OS documentation - it can be found here. I recommend reading chapter 8 about networking. In general the flow looks as presented on the pic below:

Figure 1 Packet reception diagram (from RedHat Performance Tuning Guide)

Hardware/interrupt buffers

These are internal HW buffers inside the network interface card (NIC). If you see hw buffer overruns on your system the only thing that you will know is that the transfer rate is too high for this interface. The document previously mentioned gives indication how to monitor it using ethtool and what you could do to minimize the impact.

You can monitor this using tools getting stats from the NIC like ifconfig. Below you can find an example from my Ubuntu box (overruns indicate the number of times the HW buffer overrun happened):

$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr b4:99:ba:e8:d3:6c 
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:20 Memory:d7500000-d7520000




UDP socket buffers

There is a lot explained what can be tuned in the document mentioned above. I will only try to explain how it can happen that the buffer queue will grow. In order to do that I will prepare a sample Java application (a modification of UDP echo client/server app). It consists of the server application and client one.

Test application

The client will be sending every 10 milliseconds an UDP datagram containing 64 bytes long string (16 times the word "test").

The server on the other hand will be one-thread application reading from socket. The reading thread after retrieving information from the socket will be sleeping for 1 second - 1s is set to show the effect in shorter timeframe but please notice any value above 10 ms (sleep on the client side) will cause this behaviour. In general such problem might happen in real life scenario if the same thread that is reading from socket is doing some processing.

Code for udp echo client:
package org.krystianek.udp.test.echo;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.DatagramPacket;
import java.net.DatagramSocket;
import java.net.InetAddress;
import java.net.SocketException;
import java.util.logging.Logger;
 
public class EchoClient {

          public static void main(String[] args) throws Exception {
                    String hostname = "localhost";
                    InetAddress ia = InetAddress.getByName(hostname);
                    SenderThread sender = new SenderThread(ia, 9999);
                    sender.start();
                  }

}

class SenderThread extends Thread {
          private InetAddress server;
          private DatagramSocket socket;
          private boolean stopped = false;
          private int port;

          public SenderThread(InetAddress address, int port) throws SocketException {
            this.server = address;
            this.port = port;
            this.socket = new DatagramSocket();
            this.socket.connect(server, port);
          }

          public void halt() {
            this.stopped = true;
          }

          public DatagramSocket getSocket() {
            return this.socket;
          }

          public void run() {

            try {
              String theLine ="testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttest";
              while (true) {
                byte[] data = theLine.getBytes();
                DatagramPacket output = new DatagramPacket(data, data.length, server, port);

                System.out.println("Sending : " + output.getData().length + " bytes ");
                socket.send(output);
                Thread.yield();
                Thread.sleep(100);
              }
            }
            catch (Exception ex) {
              System.err.println(ex);
            }
          }
        }

Code for udp echo server:

package org.krystianek.udp.test.echo;

import java.net.DatagramSocket;
import java.net.InetSocketAddress;
import java.net.SocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.DatagramChannel;

/**
 * @author krychu
 *
 */
public class EchoServer {

        public static void main(String[] args) throws Exception {
                DatagramChannel channel = DatagramChannel.open();
                DatagramSocket socket = channel.socket();
                SocketAddress address = new InetSocketAddress(9999);
                socket.bind(address);
                ByteBuffer buffer = ByteBuffer.allocateDirect(65507);
                while (true) {
                        SocketAddress client = channel.receive(buffer);
                        try {
                           Thread.sleep(1000);
                        } catch (Exception ex) {
                           ex.printStackTrace();
                        }
                        buffer.flip();
//                      System.out.println("Server received : " + buffer);
//                      channel.send(buffer, client);
//                      buffer.clear();
                }
        }

}

For the test the classes have been packaged into echo-0.0.1-SNAPSHOT.jar using maven build system.

Monitoring the system

The next step was to identify how to monitor the buffer utilization. There are different ways - in my article I will use the statistics available in the proc filesystem - quite good description can be found in the man page:


$ man proc
...

/proc/net/udp

Holds a dump of the UDP socket table. Much of the information is not of use apart from debugging. The "sl" value is the kernel hash slot for the socket, the "local_address" is the local address and port number pair. The "rem_address" is the remote address and port number pair (if connected). "St" is the internal status of the socket. The "tx_queue" and "rx_queue" are the outgoing and incoming data queue in terms of kernel memory usage. The "tr", "tm->when", and "rexmits" fields are not used by UDP. The "uid" field holds the effective UID of the creator of the socket. The format is:

sl  local_address rem_address   st tx_queue rx_queue tr rexmits  tm->when uid
1: 01642C89:0201 0C642C89:03FF 01 00000000:00000001 01:000071BA 00000000 0
1: 00000000:0801 00000000:0000 0A 00000000:00000000 00:00000000 6F000100 0
1: 00000000:0201 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0

...

There are two files available one for IPv4 (/proc/net/udp) and IPv6 (/proc/net/udp6) - the example output can be found below

$ cat /proc/net/udp
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops            
  564: 00000000:0801 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 13276 2 0000000000000000 0        
  601: 00000000:E026 00000000:0000 07 00000000:00000000 00:00000000 00000000   118        0 12564 2 0000000000000000 0        
...
 1833: 00000000:9CF6 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 13808 2 0000000000000000 0        
 1884: 00000000:ED29 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 1145241 2 0000000000000000 0      
$

$ cat /proc/net/udp6
  sl  local_address                         remote_address                        st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops
    2: 00000000000000000000000000000000:ADCF 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 13287 2 0000000000000000 0
   37: 00000000000000000000000000000000:9DF2 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000   104        0 10786 2 0000000000000000 0
...

 1820: 00000000000000000000000000000000:14E9 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000   104        0 10784 2 0000000000000000 0 2024: 00000000000000000000000000000000:B5B5 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 13796 2 0000000000000000 0

In addition I developed a simple ruby script for parsing the udp stats.
The full parameter set for the utility:

$ udpstat
Usage: udpstat [options]

Specific options:
    -6, --ipv6                       Enable IPv6 stat parsing
    -K, --kilobytes                  Present values in kilobytes
    -M, --megabytes                  Present values in megabytes
    -p, --ppid pid                   PID of the process to be monitored

Common options:
    -h, --help                       Show this message
    -v, --version                    Show version

    - interval in which statistics are to be reported
       - the number of statistics to be shown

@2013 Krystian Brachmanski, Wroclaw


The example dump can be found below:

$ ./measurements/perftools/udpstat -6 1 2
   Timestamp       Local IP:port        Remote IP:port    rx[B]   rbuf[B]     tx[B]   tbuf[B]   drops
  1353956875 00000000000000000000000000000000:44495 00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956875 00000000000000000000000000000000:40434 00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956875 00000000000000000000000000000000:24105 00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956875 00000000000000000000000000000000:36499 00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956875 00000000000000000000000000000000:51186 00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956875 00000000000000000000000000000000:2049  00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956875 00000000000000000000000000000000:111   00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956875 00000000000000000000000001000000:123   00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956875 000080FE00000000FFD724029C55BDFE:123   00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956875 00000000000000000000000000000000:123   00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956875 00000000000000000000000000000000:45593 00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956875 00000000000000000000000000000000:612   00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956875 00000000000000000000000000000000:5353  00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956875 00000000000000000000000000000000:46517 00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956876 00000000000000000000000000000000:44495 00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956876 00000000000000000000000000000000:40434 00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956876 00000000000000000000000000000000:24105 00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956876 00000000000000000000000000000000:36499 00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956876 00000000000000000000000000000000:51186 00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956876 00000000000000000000000000000000:2049  00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956876 00000000000000000000000000000000:111   00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956876 00000000000000000000000001000000:123   00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956876 000080FE00000000FFD724029C55BDFE:123   00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956876 00000000000000000000000000000000:123   00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956876 00000000000000000000000000000000:45593 00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956876 00000000000000000000000000000000:612   00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956876 00000000000000000000000000000000:5353  00000000000000000000000000000000:0            0  1048576        0  1048576       0
  1353956876 00000000000000000000000000000000:46517 00000000000000000000000000000000:0            0  1048576        0  1048576       0

You can find the utility in google code as part of the yapcg tool: http://code.google.com/p/yapcg/source/browse/measurements/perftools/udpstat

Testing the UDP buffers

Configuration of the test application (actually it is in the code):
- udp port: 9999
- IPv6 socket listening
- system settings:
     net.core.rmem_default = 1048576
     net.core.rmem_max = 1048576
1. Starting the Echo UDP server in the background:
$ java -cp ./target/echo-0.0.1-SNAPSHOT.jar org.krystianek.udp.test.echo.EchoServer &

2. Start the udp6 monitor the ipv6 udp sockets in the background and present the statistics in kilobytes:
$ ./udpstat -6 -KB 1 100000 > udpstat.out 2>&1 &
...

3. Starting the client:
$ java -cp ./target/echo-0.0.1-SNAPSHOT.jar org.krystianek.udp.test.echo.EchoClient

4. Analysis of the results:

- if we take a look at the generated output:
$ cat udpstat.out
   Timestamp       Local IP:port        Remote IP:port     rx[KB]  rbuf[KB]    tx[KB]  tbuf[KB]   drops
...
  1358196897 00000000000000000000000000000000:9999  00000000000000000000000000000000:0            0     1024        0     1024       0
...
  1358196898 00000000000000000000000000000000:9999  00000000000000000000000000000000:0            8     1024        0     1024       0
...
  1358196902 00000000000000000000000000000000:9999  00000000000000000000000000000000:0           38     1024        0     1024       0
...
  1358196906 00000000000000000000000000000000:9999  00000000000000000000000000000000:0           68     1024        0     1024       0
...
  1358196909 00000000000000000000000000000000:9999  00000000000000000000000000000000:0           90     1024        0     1024       0
...
  1358196910 00000000000000000000000000000000:9999  00000000000000000000000000000000:0           97     1024        0     1024       0
...
  1358196917 00000000000000000000000000000000:9999  00000000000000000000000000000000:0          149     1024        0     1024       0
...
  1358196928 00000000000000000000000000000000:9999  00000000000000000000000000000000:0          230     1024        0     1024       0
...
  1358197036 00000000000000000000000000000000:9999  00000000000000000000000000000000:0         1024     1024        0     1024       9  1358197037 00000000000000000000000000000000:9999  00000000000000000000000000000000:0         1024     1024        0     1024       8  1358197038 00000000000000000000000000000000:9999  00000000000000000000000000000000:0         1024     1024        0     1024       7  1358197039 00000000000000000000000000000000:9999  00000000000000000000000000000000:0         1024     1024        0     1024       7
...

As expected the receive socket buffer starts to increase as soon as the UDP echo client starts since the transmission rate of the client is much higher than the reception rate of the server (Server receiver once per second, while client transmitting 100 times 64 bytes per second). As soon as the receive socket buffer utilization level reaches the maximum defined in system kernel parameters one can see packets drops happening (marked with red colour).

Everything seems to be working as expected...

Conclusions

When designing an application that is processing high volumes of data always ensure that the thread reading from the socket is not doing any additional, unnecessary processing since it may cause your socket buffers utilization go high and eventually would cause packet drop. If the additional processing/decoding is necessary then still an option with a thread pool might solve the buffer issue.

Anyway in my opinion the general rule of thumb should be that it is always beneficial to make the socket reading layer as lightweight as possible. With a single threaded approach one could think of creating a queue to which the reader thread would put the payload and a pool of worker threads would read from the queue and continue with the processing.

Ubuntu: SSL/TLS handshake - determine version

Recently I had to figure out what is the SSL and TLS version supported by some framework for which the only information I had was that it supports encryption. After looking into the TLS RFC (http://www.ietf.org/rfc/rfc2246.txt) and information about SSL I figured out that one might get those information by sniffing the handshake procedure.
Having the plan I decided to first check it locally on my host. As a first step I configured the apache2 on my Ubuntu 11.04 box as described: https://help.ubuntu.com/11.04/serverguide/C/httpd.html to support encryption. Next step was to use openssl (plus wireshark to be confident;)) to get the protocol version used in the handshake procedure.

SSL 3.0

Figure 1 SSL 3.0 handshake

Now let's have a test connecting using the openssl to the local apache2 server supporting SSL 3.0.

# openssl s_client -ssl3 -connect 127.0.0.1:443
CONNECTED(00000003)
depth=0 /CN=krystianek
verify error:num=18:self signed certificate
verify return:1
depth=0 /CN=krystianek
verify return:1
---
Certificate chain
 0 s:/CN=krystianek
   i:/CN=krystianek
---
Server certificate
-----BEGIN CERTIFICATE-----
MIICpjCCAY4CCQCXhncRWkNj1DANBgkqhkiG9w0BAQUFADAVMRMwEQYDVQQDEwpr
cnlzdGlhbmVrMB4XDTExMTEwMjE2MzgyOVoXDTIxMTAzMDE2MzgyOVowFTETMBEG
A1UEAxMKa3J5c3RpYW5lazCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
ALU90KhdAKy4dBTGO8GLIpHLvO5L9DNZjLURgVoY2H8r1piYfvZzQ6sgNIrQzZjg
vGDTg2bUT7g4UQFJouSVGGRy8qk5afipoUFS+CxQEoWHkIQoRLdDSEtLxpYlQvk3
ID2qTzLGBeNm+YBSsvzg2e4sWBykKh1ePJ/TFxSdnW9+UZWbwytFlA6qNPVWA2Ks
lX4+N5xHIL1afd1+qGKUR85xJBcXXUbbZM7Fe+neGHmMqxuUXOmrU7uHE7igcHq7
UUXByIertkS33cgpV7QbWr9w+ip8ggKAUVkY/03NpaYfk40L06sef7jyWhMXFO8i
3TbnwDH0m/I3qxZqQZifYbECAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAJ4BIrQ/U
KeJWsWQJsOAfJlpQ2giUiEYLDZyP8gnmhNCYnEdCBl9ltXxPIe1RIFK/O5dnAgV/
GLPYREW+jZ4tyip4cwWMIgixtcpyWyESJlH4vF4lKj6dP5C0xP5cyfbfvR9rlNxk
eejBsUv24FK9xCPN2V8IdkS+CNneGW06yAf+EQP9mF5weNXkbE7tX+yBwKuR8BgE
AUZ1qohBPz5OZqSV61HikVUf3RmqapnIo4lejilj/1uVhU8QJ1FxwJu2UyAJaiPj
qRSVvW4H7LSkz326DaKzoFFUylhAZPImEPN0LHv1ee0yzbWAzN8iEM6bzdNYJfJ2
fGQu5yAue2vxHQ==
-----END CERTIFICATE-----
subject=/CN=krystianek
issuer=/CN=krystianek
---
No client certificate CA names sent
---
SSL handshake has read 1413 bytes and written 319 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : SSLv3
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 6C8A75631A4964A27410DF69CFD267C8C1EE6363A6FDED270BD0671B0DFAE99F
    Session-ID-ctx:
    Master-Key: FDA851ACE2E2320690D55C7766A51718FCD6B2A89ED6887A4368583D8EB2FF2D4559D2408135D60A3401711B9C5FC7A9
    Key-Arg   : None
    Compression: 1 (zlib compression)
    Start Time: 1320299309
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
---
GET /

It works!

This is the default web page for this server.


The web server software is running but no content has been added, yet.




closed
You have new mail in /var/mail/root
#


Ok, it works. In my case there was no client certificate sent towards the SSL server.


TLS 1.0

Figure 2 TLS 1.0 handshake

And now let's try to connect again using the openssl to the apache2 server but this time using the tls1 protocol.


# openssl s_client -tls1 -connect 127.0.0.1:443
CONNECTED(00000003)
depth=0 /CN=krystianek
verify error:num=18:self signed certificate
verify return:1
depth=0 /CN=krystianek
verify return:1
---
Certificate chain
 0 s:/CN=krystianek
   i:/CN=krystianek
---
Server certificate
-----BEGIN CERTIFICATE-----
MIICpjCCAY4CCQCXhncRWkNj1DANBgkqhkiG9w0BAQUFADAVMRMwEQYDVQQDEwpr
cnlzdGlhbmVrMB4XDTExMTEwMjE2MzgyOVoXDTIxMTAzMDE2MzgyOVowFTETMBEG
A1UEAxMKa3J5c3RpYW5lazCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
ALU90KhdAKy4dBTGO8GLIpHLvO5L9DNZjLURgVoY2H8r1piYfvZzQ6sgNIrQzZjg
vGDTg2bUT7g4UQFJouSVGGRy8qk5afipoUFS+CxQEoWHkIQoRLdDSEtLxpYlQvk3
ID2qTzLGBeNm+YBSsvzg2e4sWBykKh1ePJ/TFxSdnW9+UZWbwytFlA6qNPVWA2Ks
lX4+N5xHIL1afd1+qGKUR85xJBcXXUbbZM7Fe+neGHmMqxuUXOmrU7uHE7igcHq7
UUXByIertkS33cgpV7QbWr9w+ip8ggKAUVkY/03NpaYfk40L06sef7jyWhMXFO8i
3TbnwDH0m/I3qxZqQZifYbECAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAJ4BIrQ/U
KeJWsWQJsOAfJlpQ2giUiEYLDZyP8gnmhNCYnEdCBl9ltXxPIe1RIFK/O5dnAgV/
GLPYREW+jZ4tyip4cwWMIgixtcpyWyESJlH4vF4lKj6dP5C0xP5cyfbfvR9rlNxk
eejBsUv24FK9xCPN2V8IdkS+CNneGW06yAf+EQP9mF5weNXkbE7tX+yBwKuR8BgE
AUZ1qohBPz5OZqSV61HikVUf3RmqapnIo4lejilj/1uVhU8QJ1FxwJu2UyAJaiPj
qRSVvW4H7LSkz326DaKzoFFUylhAZPImEPN0LHv1ee0yzbWAzN8iEM6bzdNYJfJ2
fGQu5yAue2vxHQ==
-----END CERTIFICATE-----
subject=/CN=krystianek
issuer=/CN=krystianek
---
No client certificate CA names sent
---
SSL handshake has read 1560 bytes and written 293 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 94FA2FB475C3FA47EDE2373C610F3F3C02CB2714F7344EC433B6B8ACEDC0AC43
    Session-ID-ctx:
    Master-Key: 958861A6AC4D901FA6263C0DA92E81F9430AEB201F82032D562813309957593E9E4259F8548AFBE2CB5A7145026135F5
    Key-Arg   : None
    TLS session ticket:
    0000 - 9a c9 2b 9c e9 54 7e e4-05 de 32 40 38 0a 6d b9   ..+..T~...2@8.m.
    0010 - 6d 21 2c 2e c8 ba 6e b7-de 37 72 0f 5b 5c 69 a8   m!,...n..7r.[\i.
    0020 - da 55 4a f6 73 31 59 4c-c4 3e 37 7b 9c 87 47 97   .UJ.s1YL.>7{..G.
    0030 - 03 f4 c9 62 45 95 a9 ab-11 31 ab de bf c9 5d b4   ...bE....1....].
    0040 - 50 75 ec 6b 54 c4 05 c8-bf 44 d3 14 41 d9 ea e9   Pu.kT....D..A...
    0050 - 0a 57 c8 d1 89 4f 3b 20-c6 0b 1e f6 f4 19 af 8e   .W...O; ........
    0060 - ca f7 18 28 1c 7b c7 9f-d4 03 c1 3f bc 47 be a0   ...(.{.....?.G..
    0070 - e0 74 0c c5 57 d6 16 4b-b3 a4 f5 c7 b7 10 7e 11   .t..W..K......~.
    0080 - 03 6a 5b e6 06 aa d7 75-40 d1 fe b0 62 ae e9 aa   .j[....u@...b...
    0090 - bc 0e 2d 59 06 97 99 78-ac 69 3e 8e c4 7f 34 e8   ..-Y...x.i>...4.
    00a0 - 84 89 c3 01 13 1b 01 b2-49 21 62 b8 4b e5 93 ea   ........I!b.K...
    00b0 - 88 09 bb d2 27 d2 ab cd-b6 94 67 0e a5 9b 7c fc   ....'.....g...|.


    Compression: 1 (zlib compression)
    Start Time: 1320299341
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
---
GET /

It works!

This is the default web page for this server.


The web server software is running but no content has been added, yet.




closed
#

GTP prime library for Java (3GPP 32.295)

Recently I needed to have a client application for a GTP' external system. I did some research on the web but unfortunately did not find anything usable for me (found one java library on kenai.com but the implementation is in very early phase, the other was open-cgf written in Erlang for which I found two obstacles the language + crash dumps when it was starting up on my Ubuntu Natty 64-bit box). Having some development background from the past and having a look on the 3GPP 32.295 specification I decided that it is not a big amount of effort to implement it. I think this is a good opportunity to learn the protocol and additionally check the JBoss Netty framework - which seems to be a perfect match for such a solution. Indeed writing a client/server application using JBoss Netty is pretty straightforward (I will try to provide more details in another article) - it mainly about writing proper encoder/decoder and handlers.

The outcome can be found in the google code under the link: http://code.google.com/p/gtpprime/.

Linux: Sharing disks via iSCSI on Ubuntu 11.04

Recently I wanted to test a clustering solution that was based on a shared storage and I was looking for a solution that could work in my virtualized environment. One requirement was that the shared disk had to be visible as device (not a mounted NFS share). The choice went to iSCSI. There is quite interesting open source solution for providing NAS functionality (http://www.freenas.org/) but since it is based on FreeBSD 8.2 and my native system in Linux I would have to run in as another virtual machine - might be too much for my box (one FreeNAS virtual machine + two virtual machines hosting cluster - Figure 1). Therefore I searched for something that could be configured natively on my Ubuntu 11.04 box. What I found and decided to configure was the iscsitarget daemon - below you can the step by step instruction how to do that.

Figure 1 iSCSI client-server architecture

iSCSI Server configuration - Ubuntu

First of all you need to install the iscsitarget software - it is available in the standard Ubuntu repo - as below:

# apt-get install iscsitarget

Reading package lists... Done

Building dependency tree     

Reading state information... Done

Suggested packages:

  iscsitarget-source iscsitarget-dkms

Recommended packages:

  iscsitarget-module

The following NEW packages will be installed:

  iscsitarget

0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.

Need to get 78.6 kB of archives.

After this operation, 291 kB of additional disk space will be used.

Get:1 http://us.archive.ubuntu.com/ubuntu/ natty/universe iscsitarget amd64 1.4.20.2-1ubuntu1 [78.6 kB]

Fetched 78.6 kB in 0s (128 kB/s)   

Selecting previously deselected package iscsitarget.

(Reading database ... 315227 files and directories currently installed.)

Unpacking iscsitarget (from .../iscsitarget_1.4.20.2-1ubuntu1_amd64.deb) ...

Processing triggers for ureadahead ...

ureadahead will be reprofiled on next reboot

Processing triggers for man-db ...

Setting up iscsitarget (1.4.20.2-1ubuntu1) ...

 * iscsitarget not enabled in "/etc/default/iscsitarget", not starting...

Define the LUNs in the configuration file as follows (red colour marks the location of the LUNs):

# vim /etc/iet/ietd.conf

...

Target ubuntu.mediate:storage.sys1
        Lun 0 Path=/luns/storagelun0,Type=fileio,ScsiId=lun0,ScsiSN=lun0
        Lun 1 Path=/luns/storagelun1,Type=fileio,ScsiID=lun1,ScsiSN=lun

Create the files that will represent the LUNs. One can also use devices (e.g. USB stick) as placeholders for LUNs but for me files were just perfect - easy to move and control.

# cd /luns/

krychu@krystianek:/luns$ sudo dd if=/dev/zero of=storagelun0 count=0 obs=1 seek=10G

0+0 records in

0+0 records out

0 bytes (0 B) copied, 9.01e-06 s, 0.0 kB/s

krychu@krystianek:/luns$ ls -latr

total 8

drwxr-xr-x 26 root root        4096 2011-10-15 10:39 ..

-rw-r--r--  1 root root 10737418240 2011-10-15 10:40 storagelun0

drwxr-xr-x  2 root root        4096 2011-10-15 10:40 .

krychu@krystianek:/luns$ ls -lh

total 0

-rw-r--r-- 1 root root 10G 2011-10-15 10:40 storagelun0

krychu@krystianek:/luns$ sudo dd if=/dev/zero of=storagelun1 count=0 obs=1 seek=1G

0+0 records in

0+0 records out

0 bytes (0 B) copied, 1.285e-05 s, 0.0 kB/s

Next enable the iscsitarget in the default configuration file - modify the /etc/default/iscsitarget file's content so that it matches the one below:

# cat /etc/default/iscsitarget
ISCSITARGET_ENABLE=true

Start the iscsitarget service:

# service iscsitarget start

 * Starting iSCSI enterprise target service                              [ OK ]

                                                                         [ OK ]

Ok, that's it - the iSCSI should be configured to publish two LUNs (storagelun0 and storagelun1) from the /luns directory. Next step is to configure the client machines.


iSCSI Client- CentOS 6.0 on kvm

First of all check if the required software is installed: iscsi-initiator-utils (in my case it was). If it is not then install it from the standard repository as follows:

[root@localhost ~]# yum install iscsi-initiator-utils

...

Start and enable the iscsi and multipathd service during boot of the system:

[root@localhost ~]# service iscsi start

[root@localhost ~]# chkconfig --list iscsi

iscsi              0:off    1:off    2:on    3:on    4:on    5:on    6:off

[root@localhost ~]# chkconfig --list multipathd

multipathd         0:off    1:off    2:off    3:off    4:off    5:off    6:off

[root@localhost ~]# chkconfig --add multipathd

[root@localhost ~]# chkconfig --list multipathd

multipathd         0:off    1:off    2:off    3:off    4:off    5:off    6:off

[root@localhost ~]# chkconfig multipathd on

[root@localhost ~]# service multipathd start

Starting multipathd daemon:                                [  OK  ]

Now you can perform the discovery of the available iSCSI targets:

[root@localhost ~]# iscsiadm -m discovery -t st -p 192.168.122.1:3260

192.168.122.1:3260,1 ubuntu.mediate:storage.sys1

192.168.1.133:3260,1 ubuntu.mediate:storage.sys1

192.168.100.1:3260,1 ubuntu.mediate:storage.sys1

192.168.101.1:3260,1 ubuntu.mediate:storage.sys1

Next connect to the target. There two options you either specify the target (name, IP) and in that case the tooling will login to only this target or you leave it unspecified and you will be connected to all targets. In this manual I will use the first approach.

[root@localhost ~]# iscsiadm -m node -l -T ubuntu.mediate:storage.sys1 -p 192.168.122.1:3260

Logging in to [iface: default, target: ubuntu.mediate:storage.sys1, portal: 192.168.122.1,3260]

Login to [iface: default, target: ubuntu.mediate:storage.sys1, portal: 192.168.122.1,3260] successful.

Ok, now let's get to the multipath configuration. First of all copy the example configuration file to the /etc directory and restart the multipathd daemon:

[root@localhost ~]# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.synthetic /etc/

[root@localhost ~]# multipath -v2

Oct 15 11:01:38 | /lib/udev/scsi_id exitted with 1

Oct 15 11:01:38 | /lib/udev/scsi_id exitted with 1

[root@localhost ~]# multipath -ll

149455400000000006c756e31000000000000000000000000 dm-3 IET,VIRTUAL-DISK

size=1.0G features='0' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  `- 6:0:0:1 sdb 8:16  active ready  running

149455400000000006c756e30000000000000000000000000 dm-2 IET,VIRTUAL-DISK

size=10G features='0' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  `- 6:0:0:0 sda 8:0   active ready  running

[root@localhost ~]# ls -l /dev/mapper/total 0

lrwxrwxrwx 1 root root      7 Oct 15 11:01 149455400000000006c756e30000000000000000000000000 -> ../dm-2

lrwxrwxrwx 1 root root      7 Oct 15 11:01 149455400000000006c756e31000000000000000000000000 -> ../dm-3

crw-rw---- 1 root root 10, 58 Oct 15 09:52 control

lrwxrwxrwx 1 root root      7 Oct 15 09:52 vg_centos6hosta-lv_root -> ../dm-0

lrwxrwxrwx 1 root root      7 Oct 15 09:52 vg_centos6hosta-lv_swap -> ../dm-1

Ok, now let us configure the multipath daemon so that the device is always available as an alias (e.g. ha-mediate under /dev/mapper/ha-mediate). Add the following section to the /etc/multipath.conf file and restart the multipathd daemon:

[root@localhost ~]# cat /etc/multipath.conf
##
## This is a template multipath-tools configuration file
## Uncomment the lines relevent to your environment
##
multipaths {
    multipath {
        wwid            149455400000000006c756e30000000000000000000000000
        alias            ha-mediate
        path_grouping_policy    multibus
        path_selector        "round-robin 0"
        failback        manual
        rr_weight        priorities
        no_path_retry        5
        rr_min_io        100
    }
}
[root@localhost ~]# service multipathd restart

Stopping multipathd daemon:                                [  OK  ]

Starting multipathd daemon:                                [  OK  ]

[root@localhost ~]# multipath -ll

149455400000000006c756e31000000000000000000000000 dm-3 IET,VIRTUAL-DISK

size=1.0G features='0' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  `- 6:0:0:1 sdb 8:16  active ready  running

ha-mediate (149455400000000006c756e30000000000000000000000000) dm-2 IET,VIRTUAL-DISK

size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  `- 6:0:0:0 sda 8:0   active ready  running

[root@localhost ~]# ls -l /dev/mapper/

total 0

lrwxrwxrwx 1 root root      7 Oct 15 11:05 149455400000000006c756e31000000000000000000000000 -> ../dm-3

crw-rw---- 1 root root 10, 58 Oct 15 09:52 control

lrwxrwxrwx 1 root root      7 Oct 15 11:05 ha-mediate -> ../dm-2

lrwxrwxrwx 1 root root      7 Oct 15 09:52 vg_centos6hosta-lv_root -> ../dm-0

lrwxrwxrwx 1 root root      7 Oct 15 09:52 vg_centos6hosta-lv_swap -> ../dm-1

As you see now the device is available under it's alias - in my case ha-mediate. So that's it - now you can create a filesystem on a device, mount it and start using it ;)

If you mount the iSCSI target only from one server you can create a cluster unaware filesystem like etx3, ext4 (example below). However for granting access from multiple servers a cluster-aware filesystem has to be created.

[root@localhost ~]# mkfs.ext4 /dev/mapper/ha-mediate

mke2fs 1.41.12 (17-May-2010)

Filesystem label=

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

Stride=0 blocks, Stripe width=0 blocks

655360 inodes, 2621440 blocks

131072 blocks (5.00%) reserved for the super user

First data block=0

Maximum filesystem blocks=2684354560

80 block groups

32768 blocks per group, 32768 fragments per group

8192 inodes per group

Superblock backups stored on blocks:

    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Writing inode tables: done                           

Creating journal (32768 blocks): done

Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 34 mounts or

180 days, whichever comes first.  Use tune2fs -c or -i to override.

[root@localhost ~]# mkdir /mnt/tmp

[root@localhost ~]# mount /dev/mapper/ha-mediate /mnt/tmp/

[root@localhost ~]# ls -l /mnt/tmp/

total 16

drwx------ 2 root root 16384 Oct 15 11:06 lost+found