Wednesday, 14 November 2012

JAXB generation using gradle

Gradle XJC Ant Task

I had a problem yesterday with the gradle war task not picking up all our jaxb classes.
The build phase was correctly creating all the correct java files but wasn't including them all in the final war which was causing errors on our testing environments but not locally.
The reason they worked locally was because eclipse compiles its own classes rather than using your build tools copies.

The solution

Actually, i'll start with what the problem actually was. It turns out that gradle or one of its plugins is a bit too clever. We haven't added the generated sources to the source sets so really speaking we shouldn't have any compiled jaxb classes. However since we have a direct dependency on some of these classes it has included them for us.

The solution regardless is to add the generated-sources to the sourceSets.

Wednesday, 17 October 2012

Merge Sort

If you have an array of 8 elements, merge sort works by grouping these 8 into 4 sorted sets of 2. The sort is done through sibling pairs.

With these 4 pairs, you pair these with their siblings and take the first elements of each. Compare element 1 in both and place the lowest value in a third new array. You then compare element 1 with element 2 and place the lowest in the third array. This will go on until you are left with single pair merge sorting into a single final array.

Insertion Sort

Insertion sort works by maintaining a fully sorted list and inserting unsorted data one at a time then immediately sorting it.

So to begin with you have a single element, and that is considered sorted.
Then you add an item to the right of it. You compare these with each other and you might have to swap them. Then you have a sorted list of 2.
Then you add another item to the right of your list. You compare this with the one before it, if they're already sorted then you're done comparing for that iteration, otherwise swap them and move to the left. This goes on till you have no more data to insert.

Bubble Sort

Bubble sort is a simple sorting algorithm that compares the first two elements in the array and keeps hold of the largest, for each pass the largest will 'bubble' to the top. Each pass will require n-i comparisons.

This algorithm is considered O(N^2)

Sunday, 2 September 2012

Clojure multimethods

Defmulti

Defmulti defines a multimethod call, the methods don't need to be defined yet. It can be as simple as the following

(defmulti salutation :gender)

With this defined now, we can add conditions in the form of multimethods

Defmethod

To add a condition to this we can use the following

(defmethod salutation :f [_] "Madam")
(defmethod salutation :m [_] "Sir")

This allows us to make calls to the function and have it transparently dispatch to the correct function based on the gender keyword in the map.

(salutation {:name "Kirsty" :gender :f})
"Madam"
(salutation {:name "Josh" :gender :m})
"Sir"

Web Design

Web Dev

I've annoying been lumbered with some front-end web stuff lately. Generally, I protest vehemently at any suggestion of me doing front facing work but this task has got me a bit interested. It probably won't last.

Pseudo

There are two kinds of pseudo in CSS: pseudo-classes and pseudo-elements

Pseudo-classes

  • :link
  • :visited
  • :hover
  • :focus
  • :active
  • :first-child
  • :lang()

Pseudo-elements

  • ::first-line
  • ::first-letter
  • ::before
  • ::after

Tuesday, 21 August 2012

Classpath error snippet

File Not Found (Classpath Resource)

If this is encountered, either try modifying the classpath or prepending the file location with 'file://'

Spring Jersey Contrib

Adding Jersey to a spring webapp

Since jersey has its own servlet that would normally take the REST requests, it's difficult to wire this in with Spring beans. That's where the spring contrib library comes in. Instead of having the WEB-INF/web.xml use the servlet com.sun.jersey.spi.container.servlet.ServletContainer, you can use the below com.sun.jersey.spi.spring.container.servlet.SpringServlet.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
         id="WebApp_ID" version="2.5">

 <context-param>
  <param-name>contextConfigLocation</param-name>
  <param-value>classpath:application-context.xml</param-value>
 </context-param>

 <listener>
  <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
 </listener>

 <servlet>
  <servlet-name>service</servlet-name>
  <servlet-class>com.sun.jersey.spi.spring.container.servlet.SpringServlet</servlet-class>
  <init-param>
   <param-name>com.sun.jersey.config.feature.DisableWADL</param-name>
   <param-value>true</param-value>
  </init-param>
  <init-param>
   <param-name>com.sun.jersey.api.json.POJOMappingFeature</param-name>
   <param-value>true</param-value>
  </init-param>
 </servlet>
 <servlet-mapping>
  <servlet-name>service</servlet-name>
  <url-pattern>/*</url-pattern>
 </servlet-mapping>
</web-app>

Maven dependency

Adding this using maven would consist of the following.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
<!-- Jersey + Spring -->
<dependency>
 <groupId>com.sun.jersey.contribs</groupId>
 <artifactId>jersey-spring</artifactId>
 <version>${jersey.version}</version>
 <exclusions>
  <exclusion>
   <groupId>org.springframework</groupId>
   <artifactId>spring</artifactId>
  </exclusion>
  <exclusion>
   <groupId>org.springframework</groupId>
   <artifactId>spring-core</artifactId>
  </exclusion>
  <exclusion>
   <groupId>org.springframework</groupId>
   <artifactId>spring-web</artifactId>
  </exclusion>
  <exclusion>
   <groupId>org.springframework</groupId>
   <artifactId>spring-beans</artifactId>
  </exclusion>
  <exclusion>
   <groupId>org.springframework</groupId>
   <artifactId>spring-context</artifactId>
  </exclusion>
 </exclusions>
</dependency>

Friday, 17 August 2012

Spring web

Servlets and Naming

When defining a spring web project, requests come through to the DispatcherServlet. Therefor this must be defined in the WEB-INF/web.xml.

1
2
3
4
5
6
7
8
9
<servlet>
    <servlet-name>test</servlet-name>
    <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
</servlet>
    
<servlet-mapping>
    <servlet-name>test</servlet-name>
    <url-pattern>/</url-pattern>
</servlet-mapping>

The important thing to note with this config is the name however. The name test is used and will result in spring looking for a WEB-INF/test-servlet.xml.

Components and Config

Now that the servlet is defined, the config that goes into spring can be configured. With this example i'm using component scan to pickup beans automatically and i'm specifying the view finder which i'll explain next.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:mvc="http://www.springframework.org/schema/mvc"
 xmlns:context="http://www.springframework.org/schema/context"
 xsi:schemaLocation="http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc-3.1.xsd
  http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
  http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.1.xsd">

 <context:component-scan base-package="com.aphexmunky.web" />

 <mvc:annotation-driven />

 <mvc:resources location="/resources/" mapping="/resources/**" />

 <bean
  class="org.springframework.web.servlet.view.InternalResourceViewResolver" >
  <property name="prefix" value="/WEB-INF/views/" />
  <property name="suffix" value=".jsp" />
 </bean>

</beans>

So the InternalResourceViewResolver points to locations where spring can find files. Should a bean return a String "home", it will then as part of it's view cycle go to /WEB-INF/views/home.jsp to find the content. The jsp could be anything really but here is an example of the bean.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
package com.aphexmunky.web;

import java.util.Map;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;

@Controller
public class HomeController {
 
 @RequestMapping({"/","/home"})
 public String showHomePage(Map<String, Object> model) {
  model.put("hello", "Hello, World!");
  return "home";
 }
}

Thursday, 16 August 2012

Spring config

Properties

Using properties files is a handy way of externalizing the config without forcing people to play about with spring xml. The easiest way of dealing with this is though PropertyPlaceholderConfigurer. You can pass to it a property set of locations which then is a list of locations. See the snippet below.

1
2
3
4
5
6
7
8
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
 <property name="locations">
  <list>
   <value>classpath:warehouse.adaptor.properties</value>
   <value>classpath:warehouse.adaptor.scheduler.properties</value>
  </list>
 </property>
</bean>

These properties can then be referenced through ${}.

Wednesday, 15 August 2012

Spring

Spring Wiring

To begin wiring spring beans together we need to create an applicationContext.xml. That looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd"
 default-autowire="byType">

 <bean id="knight" class="com.aphexmunky.spring.BraveKnight"/>
 
 <bean id="quest" class="com.aphexmunky.spring.Quester" />

</beans>

With these two beans explicitly wired, we can create spring and grab our beans out of it.

1
2
ApplicationContext ac = new ClassPathXmlApplicationContext("application.xml");
Knight knight = ac.getBean(Knight.class);

Autowiring

Autowiring can be done by annotating a private member with @Autowired. For this to work however the bean still needs to be defined.

However, there is spring context which can allow us to find beans automatically. The following needs to be added to the spring config file.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:context="http://www.springframework.org/schema/context"
 xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
      http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd"
 default-autowire="byType">

 <context:component-scan base-package="com.aphexmunky.spring" />

</beans>

With this configuration, we've added spring context to the xml namespace and added context:component-scan which allows spring to find beans annotated with certain annotations and automatically consider them beans. They are

  • @Component
  • @Controller
  • @Repository
  • @Service

With these, any bean annotated with one and in the base-package specified can now be instantiated by spring.


spring xsd locations

Sunday, 12 August 2012

Refs

Updating refs

Refs, unlike atoms are a coordinated. They are however like Atoms, synchronous. All updates to refs must be done within a transaction. This can be done by wrapping the changes within a dosync block.

Alter

alter is the most like swap!. It will always be applied in the order designed and it makes no assumptions the re-orderability.

Commute

Commute is used when the order the changes are applied does not matter. It gets its name from commutative. Commute never attempts to retry and as such can be significantly faster.

Ref-set

Ref-set, like alter will retry on values. It is generally only used to to reinitialise refs state to starting values.

Validators

Validators must return true for a transactional change to take place. They can be applied using :validator fn.

The dark side of STM

The problem with all these retries that the STM provides is that it has no idea if what you're doing is unsafe. If you're writing to database, you don't want it written multiple times while another part of your application fails and the whole thing retries. For that reason there is an io! macro which throws an error if done within a transaction.

Saturday, 11 August 2012

Atoms

Practicality

Atoms are uncoordinated and synchronous reference types. Their name comes from the atomic nature. The most common way of manipulating them is through swap!.

(def sarah (atom {:name "Sarah" :age 25 :wears-glasses? false}))

(swap! sarah update-in [:age] + 3)

(swap! sarah (comp #(update-in % [:age] inc) #(assoc % :wears-glasses? true)))

Should an atom change during the swap! operation, it will automatically retry the operation. It will keep retrying until it updates undisturbed.

Reset!

reset is a way of changing an atoms value with no regard to what its current value is.

(reset! sarah :y)
:y


delays, futures and promise

Delay

Delay is a concurrency function that delays the execution of some code until it is wanted.

(def s (delay (slurp "http://squidoop.blogspot.co.uk/")))
s
#<Delay@26e7127: :pending>
@s
"<!DOCTYPE html>\n<ht...

The code takes some time to execute on the first run, however once it is realized it is cached and the next dereference will be typical execution of any string.

Future

Future is an idiomatic way of creating execution for a new thread. Unlike native Java that requires a implmentation of Runnable for a piece of code, any function wrapped in a future will create and start a new thread but return execution immediately. You can use the ref of the future straight away however it may not be fully realized when you use it.

Unlike delays however, you can specify a timeout when dereferencing a future.

(deref (future (Thread/sleep 5000) :done!) 1000 :impatient!)
:impatient!

Promise

A promise is a completely uninitialised value. You promise to deliver a value to it some point in time later.

user=> (def p (promise))
#'user/p
user=> p
#<core$promise$reify__6153@5200089: :pending>
user=> (realized? p)
false
user=> (deliver p 42)
#<core$promise$reify__6153@5200089: 42>

Combining a future promise

(def a (promise))
(def b (promise))
(def c (promise))
(future (deliver c (+ @a @b)) (println "Delivery complete!"))

The future can't deliver because both a and b are unrealised promises.

(deliver a 15)
;= #<core$promise$reify__5727@56278e83: 15>
(deliver b 16)
; Delivery complete!
;= #<core$promise$reify__5727@47ef7de4: 16>
@c
;= 31

As soon as both promises have been delivered, the future stops being blocked and the delivery execution happens.

Tuesday, 31 July 2012

Partial & Tagged Literals

Partial

Partial is a great function. The doc (doc partial) describes it as taking function f and fewer than the normal arguments to f. What is returned is a function that a variable number of remainder args. There are many potential applications and allow you to essentially create duplicate but similar functions without code duplication.

(def only-strings (partial filter string?))
(only-strings ["first test" :anothertest 3 4 "last test"])
("first test" "last test")

Tagged Literals

I mentioned a tagged literal in my last post. Namely the #inst "2012" part. That convenience is a tagged literal for creating a date. I accidently noticed there's a new one in the build i've done from github. That's the #uuid which seems to have appeared from 1.4-alpha4. It appears there aren't many available beyond that yet but they're definitely worth keeping an eye on as they're very handy. Hopefully at some point i'll do a post on creating your own.

Sunday, 29 July 2012

Focusing on Clojure

Back on Numbers

Numbers can be appended with letters that allow you to specify the type.

(type 1000)
java.lang.Long

(type 1000N)
clojure.lang.BigInt

(type 1000M)
clojure.lang.BigDecimal

Map, Reduce & Filter

These healthy staples allow you to apply functions to sequences. The usage is dependant on context. I'll go through each one:

Map

Map applies a function to a sequence of data and returns a sequence of the same length with the function applied to each

(map (fn [n] (* n 10)) '(1 2 3 4 5 6 7 8 9 10))
(10 20 30 40 50 60 70 80 90 100)

Functions with multiple arity can also be used. The result of this function is multiplying arg 1 with arg 2. This has given us a list of square numbers.

(map (fn [n m] (* n m)) (range 1 11) (range 1 11))
(1 4 9 16 25 36 49 64 81 100)

Reduce

Reduce is similar however instead of returning a list of the same size, the goal is to get to a single result. So arg 1 and arg 2 are used in combination to produce a single result, which then becomes arg 1 and #3 becomes arg 2. This is best shown with an example.

(reduce * (range 1 6))
120

To produce that output the following is equivalent:
((((1 * 2) * 3) * 4) * 5) * 6 = 120

Filter

Filter is again similar, however the size of the output is variable. You could end up with a list as long as the original or empty or anywhere in between. Filter works by providing a predicate function with the list and outputs the values of the list where the function returns a truthy value.

(filter even? (range 1 11))
(2 4 6 8 10)

ArrayList

Since clojure is on the JVM, it can work with java code seemlessly

(def x (java.util.ArrayList.))
(map #(.add x %) (range 1 11))
(true true true true true true true true true true)
x
#<ArrayList [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]>

Sequential destructuring

Vectors can be sequentially destructured as demonstrated here

(def x [5 10 [3 5] 99.9])
[5 10 [3 5] 99.9]
(let [[a b _] x] (+ a b))
15

The & can also be used

(let [[a & rest] x] rest)
[10 [3 5] 99.9]

Date

Just as a quick reference, #inst "" can be used for quick date creation

(def d #inst "2012")
#inst "2012-01-01T00:00:00.000-00:00"
(type d)
java.util.Date

Repeatedly Partial

Lookup these values

(take 15 (repeatedly (partial rand-int 60)))
(48 18 41 4 41 49 45 22 0 25 17 57 45 36 52)

Others

Some other useful functions found are memoize, doseq, dotimes, for, range, doc, find-doc and source.

Thursday, 26 July 2012

Clojure revisit

Well, it appears i've forgotton all my clojure knowledge. The only thing I appear to have remembered is the thinking style behind it all. So, after reviewing some basics today let's write them down.

Getting to the REPL

java -cp clojure-1.4.0.jar clojure.main

Basic Math

(+ 1 2 3)
(- 3 2 1)
(/ 1 2 3)
(* 1 2 3)

Numbers

Numbers can be represented infinitely appended with an M or as ratio's.
2r10110110 ; this is radix 2 (bin)
8r755 ; this is radix 8 (oct)
16rFFF ; this is radix 16 (hex)

Types

There are many forms of types available in clojure
nil ; equivalent to null essentially, it is neither true or false
true false ; these are boolean values :fred ; this is a symbol
\c \newline \space \tab ; these are chars
"string" ; this is a string literal
(1 2 3) ; this is a list
[1 2 3] ; this is a vector
{:a 1 :b 2 :c 3} ; this is a map
#{:a :b :c} ; this is a set
^{:a 1}[1 2 3] ; this is a vector with metadata

Functions

Functions are defined with the fn function, however this creates anonymous functions and to store them you need to use defn
(defn average [x y] (/ (+ x y) 2))

Useful functions

Just some I remember from last time

  • (zipmap [:a :b :c] [1 2 3])
    {:c 3, :b 2, :a 1}
  • (doc zipmap)
    ; prints current documentation
  • (source zipmap)
    ; prints source
  • (cons :y [:a :b :c])
    [:y :a :b :c]

In my next post about clojure I want to include some stuff about map, reduce and filter.

Friday, 13 July 2012

Starting an new Activity

Before starting a new activity from a current one, you need to setup the classes.

Intent display = new Intent(this, DisplayMessageActivity.class);
EditText editText = findViewById(R.id.edit_text);
String message = editText.getText().toString();
display.putExtra(EXTRA_MESSAGE, message);
startActivity(display);

From the DisplayMessageActivity then, you can get the extra like this.

Intent intent = getIntent();
String message = intent.getStringExtra(PreviousActivityClassname.EXTRA_MESSAGE);

Thursday, 12 July 2012

Android UI Weight

Android weights are like drink mixes, "2 parts vodka, 1 part coffee liqueur". If you set one widget to a weight of 2 and another to a weight of 1 - the first will get 2/3rds of the space and the other will get 1/3rd.
When using weights, it's a good idea to set the layout_width to 0 to improve performance. It improves performance because it doesn't need to work out something that ultimately is irrelevant.

Tuesday, 10 July 2012

Design Patterns

Some I can think of off the top of my head:

  • Subscriber
  • Builder
  • Singleton
  • Adapter
  • Factory
  • Facade

Patterns can be broken down into five types, four of which were described in gang of four. The fifth has been added later but its contents predate many of the other patterns. They are
  1. Creational Patterns
  2. Structural Patterns
  3. Behavioural Patterns
  4. Concurrency Patterns
  5. Architectural Patterns

Creational Patterns

So this includes Abstract Factory, Builder, Factory Method, Lazy Initialisation, Multiton Pattern, Object Pool, Prototype and Singleton

Structural Patterns

Adapter, Bridge, Composite, Decorator, Facade, Front Controller, Flyweight, Proxy and Module

Behavioural Patterns

Blackboard, Chain of Responsibility, Command, Interpreter, Iterator, Mediator, Momento, Null Object, Observer, Servant, Specification, State, Strategy, Template method and Visitor

Concurrency Patterns

Active Object, Balking, Message Design, Double-checked Locking, Event-based asynchronous, Guarded Suspension, Lock, Monitor Object, Reactor, Read-Write Lock, Scheduler, Thread Pool and Thread-Specific Storage

Architectural Patterns

SOA, MVC, N-Tier, Event Driven, Peer-to-Peer et al

As each pattern could be its own post, i won't attempt to describe them here but will work through the list as individual posts

Monday, 9 July 2012

Makefile

Before we can have a useful makefile, we should first create some files that the build tool can be run against.

module.h

#include

void sample_func();

module.c

#include "module.h"

void sample_func() {
    printf("Hello World!");
}

main.c

#include "module.h"

void sample_func();

int main() {
    sample_func();
    return 0;
}

So, now we have a directory with 3 files. A module which includes a header and source file. And a main source file that includes an entry point and has a dependency on our module. Let's make a Makefile for this.

GCC = gcc
FLAGS = -O2 -j3
CC = ${GCC} ${FLAGS}

all: main.o module.o
  ${CC} main.o module.o -o target_bin
main.o: main.c module.h
  ${CC} -I . -c main.c
module.o: module.c module.h
  ${CC} -I . -c module.c
clean:
  rm -rf *.o
  rm target_bin

Sunday, 8 July 2012

Android UI basic

Basic LinearLayout

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:orientation="vertical"
    android:layout_width="fill_parent"
    android:layout_height="wrap_content" >
</LinearLayout>


When this code is saved, the ADK will use the Android Asset Packaging Tool (AAPT) to generate R.java

In Android, a widget is called a View and a layout container like LinearLayout is called a ViewGroup. Above has created an empty ViewGroup

Asking a Question

In order to ask a question, we'll need to add a TextView to the top of the layout. To add this, put the following in between the LinearLayout element.

<TextView
    android:id="@+id/question"
    android:layout_width="fill_parent"
    android:layout_height="wrap_content"
    android:text="Please wait..."
  />

Now for the answers we're going to want multiple choices. If we use buttons for starters that'll do, but since we can't set a variable number of buttons we'll need to contain them. So let's create a new LinearLayout.

<LinearLayout 
    android:id="@+id/answers"
    android:orientation="vertical"
    android:layout_width="fill_parent"
    android:layout_height="wrap_content" >
</LinearLayout>

Then, for each button we want something similar to this:

<Button
    android:id="@+id/yes"
    android:text="Yes!"
    android:layout_width="fill_parent"
    android:layout_height="wrap_content"
/>

Saturday, 7 July 2012

Terminal Stuff

ffmpeg -i input.mp4 -c:v copy -c:a copy output.mkv

ffmpeg takes an input file, copies the codec for video and audio and sets the output file and format


openssl enc -aes-256-cbc -salt -in file.tar.gz > outfile.tar.gz

openssl set to encryption mode with aes-256-cbc encryption algorithm and salted. Sets the input file and directs the output to a file


expand -t4 Tabs.java

expand sets a number of spaces to replace tabs with and an input file, outputs to stdout


find . -iname \*\.epub -exec mv {} ~/epub/ \;

finds files using the current directory as the root, filters on the case insensitive name a regex pattern of *.epub and for each find executes mv the result to ~/epub/


nmap -sTUV -O -p- -PN 192.168.1.*

nmap set to TCP and UDP and versioning mode, also set to identify an operating system. Will scan all ports and behave as if the host is up regardless of responses. Will scan a class C network for all ip's between 192.168.1.1-255


:%s/search_regex/replacement_text/g

Vim command for finding and replacing text, the G flag represents globally across the whole file. Otherwise it will be limited to the current line


Monday, 2 July 2012

Virtual Machine Options

As Virtualisation makes it relatively easy to setup a large number of systems, it can help you configure a large number of systems. Each dedicated to a specific service. To that end, it can be separated into 5 different categories.

  • Application Level
  • Platform-level VMs
  • Paravirtualization
  • Hardware-assisted Virtualisation
  • Bare-metal Virtualisation


Application Level


Systems like WINE are application level virtualisations. They allow the installation of a single application that were designed to be run on an entirely different system.

Platform-level VMs


Applications such as VirtualBox or VMware Player are platform-level VMs. They emulate systems to allow for contained systems to have their own hardware that systems can be installed against.

Paravirtualization

While similar to Platform-level VMs, they work with fewer resources and usually require a specialised kernel such as Xen.

Hardware-assisted Virtualisation


A hardware interface where VMs have access to hardware features of the CPU such as VMX/SVM flags on Intel/AMD chips respectively.

Bare-metal Virtualisation


Some VM systems include a minimal operating system dedicated to VM operation. Two examples of this are Citrix XenServer and VMware ESX.

Notes


The KVM solution that comes with RHEL6 is known as hypervisor, a VM monitor that supports the running of multiple operating systems concurrently on the same CPU. KVM replaces the previous default of Xen.

KVM has replaced Xen in many open-source operating systems. XenSource which is owned by Citrix started working with Microsoft after RHEL5 was released.

Hard Drives and Linux (snippet)

SATA or Serial Advanced Technology Attachment


Systems now with port multipliers, can easily be configured to have 16 drives through SATA. Older systems could only handle 4 through PATA (Parallel Advanced Technology Attachment).

SCSI or Small Computer Systems Interface


Depending on the SCSI hardware, you could have up to 31 hard drive devices connected at a time.

Pick 'n' Mix


While you can combine these in as many configurations as your hardware will allow, the Linux boot files from /boot/ works on only one of the first two hard drives. If Linux is installed on the second drive, you'll need other media recognised by the BIOS/UEFI to get access to it.

The caveat to this however is that Linux doesn't have to be installed to a local drive. There are options that allow it to be placed on SAN (Storage Area Network), DASD (Direct Access Storage Device), hardware RAID devices (Redundant Array of Independent Disk) and more.

Access Control System Considerations

Before implementing any kind of access control system, the security professional needs to consider potential vulnerabilities in the proposed system. The reason for needing to consider these is because they are what give rise to threat. Threats must also be considered because they lead to risks.

Risk is the potential that a vulnerability may be exploited.

Biometrics / Biometric Authentication

Biometrics come in many forms and refers to the identification of humans by their characteristics or traits. It is used as a form of identification and access control.
They can all be broken down into two categories: Physiological or Behavioural. The latter is also sometimes known as Behaviometrics. Some examples of them are (in order of effectiveness)
  1. Palm Scan
  2. Hand Geometry
  3. Iris Scan
  4. Retina Scan
  5. Fingerprint

Effectiveness is not a sole factor to consider when choosing biometric systems however. There are many more things that need to be weighed in. Performance is generally measured by the following methods:
  • False Accept Rate (FAR) or False Match Rate (FMR)
  • False Reject Rate (FRR) or False Non-Match Rate (FNMR)
  • Relative Operating Characteristics (ROC)
  • Equal Error Rate (EER) or Crossover Error Rate (CER)
  • Failure to Enrole Rate (FER)
  • Failure to Capture Rate (FCR)
  • Template Capacity

Android Services

A Service is a process that runs in the background, without interacting with the user. Android provides some pre-defined services which can be accessed through getSystemService().

To declare your own service, it must be defined in the manifest as a service and the implementing class must extend Service or one of its subclasses. A service will not automatically run in its own thread without the process attribute. They by default run in the main thread of the hosting process. To define the process, in the manifest set the process under the application tab. An example is :my_service. The colon at the start tells android that the process is private to declaring applications. Otherwise the process would be globally accessible.

An Activity can start the Service by calling the startService() method and stop the service via the stopService() method. After a service is started the onCreate() method is called and later the onStartCommand gets invoked with the Intent.

Starting services automatically


To start services automatically after boot involves registering a BroadcastReceiver to the Android android.intent.action.RECEIVE_BOOT_COMPLETED system event. This requires the RECEIVE_BOOT_COMPLETED permission. In the broadcast receiver class, override the onReceive() method and call the service like:

Intent service = new Intent(context, MyService.class); context.startService(service);

Related Information

Android Networking
Android Services

MBeans

Standard MBeans

A standard MBean is created by creating an interface that complies with the naming convention of SomethingMBean. The implementation is then called Something. A NotCompliantMBeanException exception will be thrown if this convention is not followed.
Following the naming convention, the standard JavaBean conventions can be followed in terms of getters and setters.

Remote Monitoring

Once the resource has been created by MBeans, the management of that resource is done by a JMX agent. The core component of the JMX agent is the MBean server. The MBean server is a managed object server in which MBeans are registered. A JMX agent also includes a set of services used to manage MBeans.
To create a basic JMX Agent, the following is minimal

MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
ObjectName on = new ObjectName("package.location.of.mbeans:type=Example");
Example e = new Example();
mbs.registerMBean(e, on);

Provided the application remains running, you can then fire up jconsole and monitor the process within there. Under the MBeans tab you should then find package.location.of.mbeans with a unit below it called Example which can then be observed, called, monitored etc.

Warning

I had some issues getting the small example working on a windows machine. This ended up being a combination of rights access (which I was struggling with on windows) and folders that existed in %TMP%. They had my username including capitals which is correct as far as I know, but they needed deleting and when they were recreated with lowercase they worked fine. You can see evidence of this by having jconsole grayed out on its monitoring processes. There is more on this here.

I also had issues where i couldn't connect to the weblogic process. This turned out to be a classpath problem which can be solved by using the jconsole of weblogics jrocket jvm. More than on that here

MXBeans

An MXBean is a type of MBean that references only a predefined set of data types. This allows you to be more flexible in allowing your clients (including remote clients) to use your MBean without allowing them access to more data than they need. The process for creating them is similar to MBeans.

First you must create an interface that either follows the naming convention of SomethingMXBean, or annotate the interface with @MXBean. Then have a class implement that interface. Unlike MBeans, there is no requirement on the implementation to be called Something.

The idea behind MXBeans is that types such as MemoryUsage that are referenced in the MXBean interface MemoryMXBean are mapped to a set of types known as Open Types that are defined in the package javax.management.openmbean. The general principle is for simple types such as int or String to remain unchanged, while complex types such as MemoryUsage get mapped to the standard type CompositeDataSupport.

MXBeans Example

To create an MXBean interface, almost exactly the same as an MBean example

public interface QueueSamplerMXBean {
    public QueueSample getQueueSample();
    public void clearQueue();
}


Then the implementation:

public class QueueSampler implements QueueSamplerMXBean {

    private Queue queue;

    public QueueSampler(Queue queue) {
        this.queue = queue;
    }

    public QueueSample getQueueSample() {
        return new QueueSample(new Date(), queue.size(), queue.peek());
    }

    public void clearQueue() {
        queue.clear();
    }
}


Then unlike MBeans, we need an additional class for the type.

public class QueueSample {

    private final Date date;
    private final int size;
    private final String head;

    @ConstructorProperties({"date", "size", "head"})
    public QueueSample(Date date, int size, String head) {
        this.date = date;
        this.size = size;
        this.head = head;
    }

    public Date getDate() {
        return date;
    }

    public int getSize() {
        return size;
    }

    public String getHead() {
        return head;
    }
}


The MXBean framework calls all the getters in QueueSample to convert the given instance into a CompositeData instance. It uses the @ConstructorProperties annotation to reconstruct a QueueSample instance from a CompositeData instance.

Registering the MXBean is much the same as registering an MBean, however in this case the difference is we need to construct the queue system before we can use it. We register the QueueSampler class against the MBean server and don't actually reference the returned type in our calling code.

Notifications

The model for adding notification support is actually quite simple. When you create the implementation of your MBean, you will want to extend somewhere in your hierarchy the class NotificationBroadcasterSupport and where you want to create notifications simple instantiate a subclass of Notification such as the pre-existing AttributeChangeNotification. Once constructed it can be passed to sendNotification().

After adding this support, you will have the ability to subscribe to notifications from tools such as jconsole.

Related resources

Oracle JMX Trail
wiki MBeans
MBean Server interface

Thursday, 28 June 2012

Multi-factor Authentication

There are three regulatory approved factors of authentication, all forms of authentication can be grouped under one of the following:
  • Something you know
  • Something you have
  • Something you are
Two-factor authentication is when two of those three authentication types is used in conjunction with each other. The most commonly seen use of this is debit/credit cards(something you have) with a pin(something you know).

Something you know

The weakest form of the three and the most commonly seen in the form of passwords or pins. One of the most troublesome issues with this form is that there is no regulation around how it is kept secure. A password can be shared or exposed through many methods without the keepers knowledge. Often there are technical constraints imposed on the creators of passwords which limits their entropy and the keeper needs to be able to remember them which makes it easy for them to be guessed.

Something you have

This form of authentication has been around for centuries, most commonly seen in the form of a key to a lock. The formal description for this is that the key embodies a shared secret between the lock and the key.
There are four ways of attacking such a system:
  • Attack the authenticator or management system to try determine the secret.
  • Steal the 'something you have'
  • Make a copy of the 'something you have'
  • A man-in-the-middle attack where the attacker sits in between the communication channel of each entity

Something you are

The strongest factor of authentication and typically seen in the form of biometrics, something you are can compose of many things such as finger prints, iris scans, voice patterns etc. They are very susceptible to replay attacks.

Tuesday, 26 June 2012

Cookies

Cookies were originally designed to add persistence to the state, a website or user had taken in the past. Although cookies can not contain virus' or malware. In the modern web environment, cookies perform several critical functions. The most common of which is authentication cookies, used to know if a user is logged in or not.

The current specification (October 2011) for cookies is rfc6265

The types of cookies

  • Session Cookie
  • Persistent Cookie (also known as tracking cookies)
  • Secure Cookie
  • Http Only Cookie
  • Third Party Cookie
  • Super Cookie
  • Zombie Cookie
Session Cookie
This type only lasts for the duration the user is on the website. To be a session cookie, no 'Expires' directive is provided when it is created.
Persistent Cookie
This type will outlast user sessions and has a max age of 1 year. Each time the site is visited the initial value set in this cookie will be provided. This could contain how the user originally came to the site, which is why it is also known as a tracking cookie.
Secure Cookie
This type has a secure attribute enabled and can only be used through HTTPS, ensuring encryption when it is transmitted between the client and the server.
Http Only Cookie
Supported by most modern browsers, this type is transmitted only when transmitting HTTP/HTTPS requests. With this, the access by client side javascript or other non-html code is mitigated. This does not totally eliminate the risk through XSS.
Third Party Cookie
This type are ones that have the domain set to a different one to the address being visited. Advertisers use this commonly to track a users browsing history.
Super Cookie
This type has a domain set as merely a top-level suffix, such as .com - security settings usually prevent this type.
Zombie Cookie
This type is recreated by other means automatically after the user has deleted it. This could be done through a variety of ways such as local flash storage.

A web browser is generally accepted to be able to store at least 300 cookies of 4Kb each and at least 20 cookies per server or domain.

Monday, 25 June 2012

Three Types of Access Control

Administrative

These security controls are put in place to define and guide employee actions in a workplace when dealing with sensitive information. E.g. Policy might dictate that HR do background checks on all employees with access to sensitive information.

Technical

These are devices, processes, protocols and other measures used to protect C.I.A of sensitive information. These might include logical access systems (Lock and key), encryption systems, antivirus systems, firewalls etc

Physical

Security controls are devices and means to control physical access to sensitive information and its availability. Examples could be a physical access systems (fence, guards etc), physical intrusion detection systems (motion sensor, alarms etc) or physical protection systems such as fire alarms, backup generators.

Saturday, 23 June 2012

RSA and DSA

DSA (Digital Signature Algorithm) is an Asymmetric encryption algorithm that is known for being fast at cryptographically signing but slow at verifying. Slow at encrypting but fast at decrypting.
RSA (named after its published creators Rivest, Shamir and Adleman) and is also an Asymmetric encryption algorithm.

To define the difference between Asymmetric and Symmetric encryption: Symmetric relies on a single private key mixed with secret input - you can use block or stream ciphers. Asymmetric uses a combination of a public and private keys and require fixed bit lengths such as 1024, 2048 etc. The public key is allowed to be distributed so long as the private key remains secure. The private key is used for digitally signing where the public key can be used to verify the integrity of that.

The encryption is done through the intended recipients public key and the creators private key, then the combination of the recipients private key and senders public key can decrypt it.

DSA is based on a discrete logarithmic problem. RSA is based on the difficulty in the factorization of large integers.

RSA is far more common and commercially accepted. While it is possible for DSA to be over 1024 bits, many applications limit it to this. As such 2048 RSA algorithms are widely accepted as the minimum security standard to be cryptographically secure.

Friday, 22 June 2012

Android SSL/TLS

The javax.net.ssl packages provide all the necessary classes and interfaces required to program secure socket abstractions based on the SSL protocols SSLv3.0 / TLSv1.1.
All of the details of the SSL handshake protocol are accounted for and the client or a server can specify the cipher set to use. X.509 Certificates are verified and if desired, the client and server each have the option of verifying the entire certificate chain until the root Certificate Authority is reached.
Android uses code from the Legion of the Bouncy Castle and OpenSSL.

Defence in Depth

Defence in depth is a security and information assurance concept in which multiple layers of security controls are placed throughout an IT system. Its intent is to provide redundancy should the event of a security control fail or a vulnerability is exploited. So should a system fail or the bad guy managed to defeat it, there would still be a variety of fallbacks to protect the network. Firewalls for homes are not as popular as they used to be due to the increase use of home routers which act as a hardware firewall for all inbound connects unless outbound traffic to that destination has preceded it.

Defence in depth measures should not only prevent security breaches but also buy an organisation time to detect and respond to an attack, thereby reducing and mitigating the consequences of attack.

Thursday, 21 June 2012

VPN Basics

a Virtual Private Network is a technology that provides secure communication through an insecure and untrusted network (like the internet). It achieves this through authentication, encryption, compression and tunneling.

What is tunneling?

Tunneling is a technique that encapsulates packet headers and data within the payload of another protocol. This way the packet can traverse the network in a manor it would otherwise not be capable of traversing.

How is it done?


The most common ways of creating a VPN are IPsec and SSL/TLS.

IPsec



IPsec provides authentication,encryption and compression at the network level. IPsec is actually a suite of protocols developed in the mid 90's and supports both IPv4 and IPv6. However it is mandatory in IPv6 and optional in IPv4. To implement IPsec two new protocols were added: Authentication Header (AH) and Encapsulating Security Payload (ESP). Handshaking and exchanging session keys are done using the the Internet Key Exchange (IKE) protocol.
The Authentication Header is protocol number 51 and it authenticates both the header and the payload. The AH however does not use encryption so it is almost never used.
Encapsulating Security Payload is protocol number 50. It enables you to add a security policy to the packet and optionally encrypt it. The encryption is done in the kernlet via the cryptoAPI. When two machines are connected via the ESP protocol, a unique number identifies the connection. This number is called the SPI (Security Parameter Index). Each packet contains a sequence number and a checksum which is called the ICV (Integrity Check Value). The checksum is calculated using a secret key which is known only to these two machines.

IPsec has two modes: Transport and Tunneling. When creating a VPN, we use tunnel mode. This means that each IP packet is fully encapsulated in the new IPsec packet. The payload of this new packet is the original packet before it was encapsulated.

The problem with this model is that in networks where the peer is behind a NAT (Network Address Translation) device. Using a NAT is a common way of connecting machines that are not directly accessable to the outside world. The NAT is performed on a machine that does have access, this is usually a gateway. The NAT modifies the IP packet and as a result the peer rejects it because the signature is wrong. The solution is commonly known as NAT-T (Network Address Translation Traversal) and works by encapsulating the IPsec packet in UDP packets so that they'll be able to pass through the NAT routers without being dropped.

OpenSwan is an open-source project that provides an implementation of IPsec. The SWAN name comes from Secure Wide Area Network which is actually a trademark of RSA. OpenSwan supports Opportunistic Encryption (OE), which enables the creation of IPsec based VPNs by advertising and fetching public keys from a DNS server.

SSL/TLS


OpenVPN is a VPN solution based on SSL/TLS. It is simpler in comparision with IPsec and OpenVPN supports RSA authentication, Diffie-Hellman key agreement, integrity checks etc. When running in server mode, it can support up to 128 clients over the same port. You can setup your own Certificate Authority and generate certificates and keys for an OpenVPN server and multiple clients.

OpenVPN operates in user-space mode; this makes it easy to port to other operating systems.

Overview of the SSL or TLS handshake

Secure Sockets Layer and its successor Transport Layer Security are cryptographic protocols that provide security over the internet.

They encrypt the segments of network connections at the Application Layer for the Transport Layer using Asymmetric Encryption for key exchange and Symmetric Encryption for confidentially and integrity.

Most protocols can be used in conjunction with TLS. Often this is done by specifying a different port, 443 for HTTPS. Another way of achieving the TLS connection is by the client requesting that the server switch to TLS. This is usually done by a protocol specific mechanism.

Once an agreement between A and B has been made to use TLS, a stateful connection is established by using a handshake procedure. During this handshake various parameters are passed between each other to establish the connections security.
  1. The client sends the server the clients SSL version number, cipher settings and other details it needs to establish a connection
  2. The server does the same back however it also sends it's own certificate (The certificate contains the servers public key) and if the client is asking for a resource that requires authentication, the server asks for the clients certificate
  3. The client uses what it has been provided to authenticate the server. If the server can't be authenticated then the user is warned that the secure encryption connection can not be established.
  4. Using all the data generated in the handshake thus far, the client creates the pre-master secret for the session and encrypts it with the servers public key. It sends the encrypted pre-master secret to the server
  5. If the server has requested client authentication, the server attempts to authenticate the client. If the client can not be authenticated the session ends. If the client can be authenticated, it uses its private key to decrypt the pre-master secret.

Both client and server have now established that they trust who each other is and are both aware of the pre-master secret. With this they both perform a series of steps to establish the master secret.

Both parties now use the master secret to generate session keys, which are symmetric keys used to encrypt and decrypt information exchanged during the SSL session and to verify its integrity.

The client now sends the server a message to state that future messages will be encrypted with the session key along with an encrypted message to state that it has finished it's own portion of the handshake.

The server performs the same action.

Now that the connection is established, the data shared between them is encrypted and decrypted. It's worth noting that at any time either side may renegotiate the connection and the whole process is repeated.

Applications

In application design, TLS is usually implemented on top of the transport layer protocols. Encapsulating the application specific protocols such as HTTP, FTP, SMTP, NNTP and XMPP.
TLS has also been implemented on datagram oriented transport protocols such as UDP (User Datagram Protocol) and DCCP (Data Congestion Control Protocol). This usage has been standardised as DTLS (Datagram Transport Layer Security).

TLS can also be used to tunnel an entire network stack to create a VPN (Virtual Private Network), as is the case with OpenVPN.
TLS is also a standard method to protect SIP (Session Initiation Protocol) application signaling. TLS can be used to provide authentication and encryption of the SIP signaling associated with VoIP and other SIP based applications.

Related Resources

Related Wikipedia Reading

TCP/IP

I'll try to get down some of the main points of TCP/IP here.
Transmission Control Protocol / Internet Protocol is a stack of protocols defined for networking and exchange of data between computers. While the suite isn't limited to TCP, it's so common that that's how it's known.

The reason I referred to the suite as a stack is because they have a layered structure with each layer looking after certain aspects of data transfer. The entire stack can be broken down to:
  1. Link Layer (Hardware / Ethernet)
  2. Protocols Include:
    • ARP
    • RARP
    • PPP
    • Ether
  3. Network Layer (The Invisible layer)
  4. Protocols Include:
    • IP
    • ICMP
  5. Transport Layer
  6. Protocols Include:
    • UDP
    • TCP
  7. Application Layer (The Visible Layer)
  8. Protocols Include:
    • Actual running applications. FTP client, browser etc
  9. Physical Layer (Not part of TCP/IP)
  10. Protocols Include:
    • Telephone line

The stack has an almost cyclic attribute about it. Data travels from the Link Layer to the Physical Layer at the source and gets picked up at the Link Layer at the destination. Once there, each layer can hand off to the next.

There is obviously a lot more going on here and to understand the next point we need to delve into issues than can occur during data transfer using this model. The main problem is that the following two scenario's are likely to happen.
  • Data Corruption
  • Data Loss

Data Corruption

So data corruption is where the data arrived at the destination but it isn't what we sent. This can happen for a number of reasons but as an example, lets say the telephone line the message is sent from isn't very well shielded from interference.

Data Loss

Our data didn't even arrive - we don't need to know where it went, just that we didn't get it.

Overcoming these issues

So the solutions to these problems is actually quite easy. For the first issue of corruption, a simple checksum can be used to validate the content. For the second issue of data loss, sequencing numbers are used to order the packets. This makes it easy to see when we are missing data.

Checksum

So what is the checksum? This is (typically) a 16 bit value of the sum of all the octets in the datagram.

Sequencing

Because both of our issues are common, data is split into small packets and each is sent individually. This allows for a lot of flexibility in our error handling. These packets of data are not guaranteed to arrive in the order they were sent. To assist with realising if piece 4 has arrived after piece 5, a sequence number is used for each packet. This allows the receiver to order their packets in the order they were intended to be.

There is another way of detecting data corruption:

Handshaking

Handshaking is a series of messages that are sent back and forth. This takes the form of a SYN, SYN-ACK and ACK messages. Which are synchronise, synchronise acknowledgement and acknowledgement respecting. To put this down to basics, message 2 won't send until we get a message 1 acknowledgement. If the acknowledgement isn't received then a time-out occurs and message 1 would be resent.
So, what if message 1 was received but it was corrupted and didn't pass the checksum? The integrity would still be maintained if that packet was just discarded and the time-out waited for, but that is terribly inefficient. Instead a NACK message is sent instead of an ACK message.
An ACK message of 1000 would mean that all data up to octet 1000 had been received thus far.
So let's take an example of sending an email and use the knowledge we've just learned. Email's are most commonly sent over a protocol known as SMTP or Simple Mail Transfer Protocol. This protocol lives on the application layer. Say we have a SMTP server sitting somewhere that we need to pass our email on to. For the application to send this message it needs to pass its data the TCP protocol which belongs to the transport layer. This then needs to pass over to the IP protocol on the network layer which holds the checksum and sequencing features.

Three-way handshake

As part of the three way handshake, certain information is gained which gathers the information required to transmit with a minimum of problems. Information about the IP address, port number and datagram sizes are all collected at this point. Once the maximum datagram size is known the transmission can be split up. Each datagram has its own TCP header and includes the source and destination port, sequence number and checksum. TCP itself supports full duplexing, which means that data can go both upstream and downstream at the same time.

TCP supports a number of flags, we've already seen SYN, SYN-ACK and ACK. There is also:
  • RST - Reset
  • PSH - Tells the application to pass all queued data
  • FIN - closes the connection

User Datagram Protocol

UDP is a member of the transport layer mentioned at the start. It is quite a different beast to TCP however and serves very different uses. Sometimes a datagram doesn't actual need to be split up and a single one will do just fine. UDP doesn't follow the same pattern of TCP, there's no synchronisation involved or sequence numbers and there doesn't even need to be any acknowledgement from the destination. Broadcasting is also possible with UDP.