Greece's Public Sector merges its data, how it was done

Kallikratis is the codename of the largest project ever conceived in the later years (2010-2011) of Greece's public sector digitalisation / computerisation effort and was founded by the Ministry of Interior and Decentralization.

The goal? To migrate different datasets spread among different applications and databases within the public sector's infrastructure.

That accounts to million rows of citizens' and public organizations' data / datasets distributed among different applications, structures, databases etc. having to be migrated in a homogenous, valid, normalized and commonly accepted data structure. In other words an ETL - Extract Transform Load process was required.

imho the talks and negotations to define the details of the final homogenous structure for each dataset was the hardest part, requiring cooperative thinking and strategizing among different tech companies and the ministry.

Datasets were categorized by domain: for example citizens' demographic data, public sector's and prefectures' economical/budgeting data, document management systems' data, and the list goes on.

The final process took place in the beggining of the year and lasted about a month, the merge happened in the first five days with corrections following until the end of January.

The project involved 2 vital factors:

1. had to be fast - ( execution, implementation,debugging, ability to change how things work on site wihtout the need to alter code / or recompile enabling tech support to operate)
2. and opensource

The plan was to extract and transform the data to a common XML file, validate it against the corresponding XSD and finally load it into the new environment. We ended up with a set of numerous transform processes and one generic load process that accepted as input the transformed xml files. 

Codename of the process was Datapair,  thumbs up for the...original name i found are welcome - sarcasm ensues :)

Various apis / platforms were investigated with some of them being:

• Pentaho Data Integration , (PDI)
• Clover ETL
• Talend Open Studio

For reasons that i would not like to present here and now, we chose the pentaho solution.

(briefly, CloverETL came at a cost for advanced features, while Talend's performance was found subpar 

at least at that time)

In the end everything worked better than expected and i am especially  happy about our tech support department beeing able to learn and finally operate the Pentaho Data Integration suite. That was a great relief enabling more people to take part in the process.

Someone could consider this,(and i mean the project as a whole from conceptualization to implementation), a default practice or a standardized ,(sic), solution but for Greece's Software industry was well...groundbreaking.

Expect more articles to come,  further analysing Pentaho's components as a minimal tribute to the platform that did the job for us.

Extra thank you for reaching the end of this post :)

JSF and Seam Security - custom login in 3 steps

Seam framework provides a standard IdentityManagement API to manage user management and authentication.The Credentials class includes the two basic properties needed to perform user authentication. The username, and the password, (sic).

But some times extra information is needed to authenticate a user. For example a company or an organization idenntification number, etc.

What we can do is simply extend the Credentials class including these properties.In the following scenario an extra company field is added to the login page.

1. Create a new class - CustomCredentials - extending Credentials, which includes the private Company property.

/*
 *@(#)CommCredentials.java  1.00  26/05/2011
 *
 */

package com.comm.security;

import com.comm.entities.Company;
import org.jboss.seam.ScopeType;
import org.jboss.seam.annotations.Install;
import org.jboss.seam.annotations.Name;
import org.jboss.seam.annotations.Scope;
import org.jboss.seam.annotations.intercept.BypassInterceptors;
import org.jboss.seam.security.Credentials;

/**
 *
 * @author Charis Arapidis - arapidhs@gmail.com
 * @version 1.00, 26/05/2011
 * @since v1.00
 */
@Name("org.jboss.seam.security.credentials")
@Scope(ScopeType.SESSION)
@Install(precedence = Install.APPLICATION)
@BypassInterceptors
public class CommCredentials extends Credentials {

    /** */
    private Company company;

    /** @return the company */
    public Company getCompany() {
        return company;
    }

    /** @param value the company to set */
    public void setCompany(final Company value) {
        this.company = value;
    }

}

Notice that CustomCredentials is a Seam Component since it extends Credentials and is installed at Application Scope.

2. In the login page add the Company field refering to it with the expression #{credentials.company}

<h:outputLabel for="company" value="#{messages.login_company}:"/>

<h:selectOneMenu id="company" value="#{credentials.company}">
    <s:selectItems value="#{CompaniesList.companies}" var="company"
        itemLabel="#{company.title}"
        noSelectionLabel="#{messages.general_select}"/>
    <f:converter converterId="CompanyConverter" />
</h:selectOneMenu>

<h:commandButton action="#{identity.login}"/>

3. Finally, in the Authenticator class inject the CustomCredentials comopent created earlier providing access to the user submitted Company field.

/*
 *@(#)Authenticator.java  1.00  11/05/2011
 *
 */
package com.comm.security;

import com.comm.entities.Company;
import com.comm.entities.User;
import com.comm.sessionbeans.SecuritySessionBeanLocal;
import org.jboss.seam.ScopeType;
import org.jboss.seam.annotations.In;
import org.jboss.seam.annotations.Name;
import org.jboss.seam.annotations.Out;
import org.jboss.seam.annotations.Scope;
import org.jboss.seam.international.StatusMessages;

/**
 * @author Charis Arapidis - arapidhs@gmail.com
 * @version 1.00, 11/05/2011
 * @since v1.00
 */
@Name("authenticator")
@Scope(ScopeType.EVENT)
public class Authenticator {

    @In
    private CustomCredentials credentials;
    /** */
    private User loginUser;
    /** */
    @In(create=true)
    private SecuritySessionBeanLocal service;

    /** @return true if user is authenticated */
    public boolean authenticate() {
        
        final String username = credentials.getUsername();
        final String password = credentials.getPassword();
        final Company company = credentials.getCompany();

        final boolean login = service.authenticate(...);

        if (login) {

        }

        return login;
    }

    @Out(value = "loginUser", required = false,
    scope = ScopeType.SESSION)
    public User getLoginUser() {
        return loginUser;
    }

    public void setLoginUser(final User value) {
        this.loginUser = value;
    }

}