First Flexbox Taste

The main feature of my CAMP project for monitoring performance of charging system, I should show multiple charts of different data sources and statistics in a a page view-port (it supposed to be used in NOC center on TV). So I decided to  try flexbox to learn something new and use new technology in my project. Dashboards can be defined to have different rows and columns, but my default dashboard is 3x3, so don't want to waste time more, it is the final

See the Pen epRoQw by Omid Mehdizadeh (@omidmt) on CodePen.

The first level of flex(1 column of 3 rows) is applied by second-content class and next level 3 columns inside each row is applied by frow class.

Challenges:

• Heights as visible part, not more:
• It is fixed by height: 97vh to body and max-height: 100% and min-height: 100% to main content div.

• IE support of flex-basis:
• It is solved by browser specific flex attribute as -ms-flex: 1 1 30vh and -ms-flex: 1 1 30%

Expiring AES Encrypted Message

As told in previous post I implemented some kind of PGP for encrypting sensitive information like login username and password. So first use openssl friendly AES to encode login information, then encrypt AES random key and salt with RSA public key, in page side, then reverse it on server. So they can ensure the information cannot be decrypted easily by man in middle (I know using https may help a lot for such things, but I liked to add my security layer there to have full control on what happened).

But another challenge appeared that was what happen if a man in middle get the ciphered information including the encrypted message and encrypted key and use them to submit, server validate and it works, while user hasn't changed its password. Another thing that got on my nerve was the hard-coded string of openssl on AES message (Salted__). So combination of these two led me to replace the fixed string by the timestamp.

Now I have the the time AES key is generated and I can compare it with the server time and if it pass a threshold, assume message as expired and leave it. It is still not a certain way to prevent man in middle completely, but with a good threshold can mitigate the risk. I chose 3610 second, 1 hour because of day light saving and 10 seconds for network and computing latency.

So the timestamp is added as part of the salt and 'Salted__' prefix is replaced by that time in encoded format. The format of message will be as:

8 bytes encoded timestamp + 8 bytes random salt + encrypted message

In this case even if the message prefix time is changed to show not expiry, the decryptor can detect it and not decrypt it, since that timestamp is used as part of encryption salt and make key faulty. That's my PGP implementation that I called OPGP or Omid PGP.

Cons:

As mention it depends on client system time, so if it is wrong, it never can get through, or I put 1 hour in my threshold to support daylight saving mismatch between client and server that increase the risk of hijacking information. For this one I'm going to test existence of a time variable that supposed to be set by server on page and compare with local time, then use up to dater one.

Grails-JavaScript Encrypted Communication - Part II

In previous post told about setting up grails environment to use bouncycastle lib. Now move to the page side and javascript. After searching around for having a comprehensive, well-documented and simple to use js lib for RSA and AES, I found pidCrypt interesting (because of the live demo page and simple instruction to use and mainly PEM support, that after going to implement, I found out that is not supported, but I copied the following function from demo page to use it).

function certParser(cert){
    var lines = cert.split('\n');
    var read = false;
    var b64 = false;
    var end = false;
    var flag = '';
    var retObj = {};
    retObj.info = '';
    retObj.salt = '';
    retObj.iv;
    retObj.b64 = '';
    retObj.aes = false;
    retObj.mode = '';
    retObj.bits = 0;
    for(var i=0; i< lines.length; i++){
        flag = lines[i].substr(0,9);
        if(i==1 && flag != 'Proc-Type' && flag.indexOf('M') == 0)//unencrypted cert?
            b64 = true;
        switch(flag){
            case '-----BEGI':
                read = true;
                break;
            case 'Proc-Type':
                if(read)
                    retObj.info = lines[i];
                break;
            case 'DEK-Info:':
                if(read){
                    var tmp = lines[i].split(',');
                    var dek = tmp[0].split(': ');
                    var aes = dek[1].split('-');
                    retObj.aes = (aes[0] == 'AES')?true:false;
                    retObj.mode = aes[2];
                    retObj.bits = parseInt(aes[1]);
                    retObj.salt = tmp[1].substr(0,16);
                    retObj.iv = tmp[1];
                }
                break;
            case '':
                if(read)
                    b64 = true;
                break;
            case '-----END ':
                if(read){
                    b64 = false;
                    read = false;
                }
                break;
            default:
                if(read && b64)
                    retObj.b64 += pidCryptUtil.stripLineFeeds(lines[i]);
        }
    }
    return retObj;
}

On server side, when I tried to decrypt the message I found 2 level bas64 encoding on the message. So first decode as base64 the hash and pass it to doFinal and again decode result as base 64 to have byte [] of the message, then convert to string.

Cipher decryptRSACipher = Cipher.getInstance("RSA/ECB/PKCS1Padding", "BC")
decryptRSACipher.init(Cipher.DECRYPT_MODE, privateKey)
new String(decryptRSACipher.doFinal(hash.decodeBase64()).decodeBase64(), StandardCharsets.UTF_8)

It supposed to be good thing for form submission from javascript side.

The next challenge was decrypting the actual data that encrypted by AES, but the decrypted key in previous by RSA didn't work, so after some research on source and net, another door of knowledge is opened to my eyes. As you may know, it is recommended to use a salt as part of AES key (password), that is fine, but in openssl implementation (that pidCrypt AES follow), the salt is send as part of prefix to the encrypted message (byte 8 to 16), and interesting thing is first 8 bytes are assigned for a "Salted__" string, yes a fixed string. So the message that supposed to be decrypted should be extracted according to the following format.

"Salted__" + 8 Bytes Random Salt + EncryptedMessage

I did some changes to this format to support one time password or expiring the key that will be explained in next post.

This link may describe openssl AES format better than me.

Grails-JavaScript Encrypted Communication - Part I

For adding new feature to my framework RIRA I should have encrypted communication between client (page) and server(controller) for sensitive information and actions like username and password on login or setting new password. So the basic plan was having RSA (private/public key) encryption of form data in page by javascript and decrypt and use them in controller side.

On testing, troubleshooting and studying best practices for this purpose I found that RSA supposed to be used for encrypting short length data, something like a password or key and not the whole form data. So the best practice was encrypting data by AES with a random key, then encrypt that random key with RSA public, and do vice versa on server side.

For the server side I chose the bouncycastle java lib as provider since it is supporting PEM key reading (parsing), because I want to make it easy for the admin of the application to generate and change the key simply, by something like openssl command line in this case. Here is the method to read PEM file:

static KeyPair getKeyPair(String pk, char [] password) {
 PEMParser pemParser = new PEMParser(new CharArrayReader(pk.toCharArray()))
 Object object = pemParser.readObject()

 PEMDecryptorProvider decProv = new JcePEMDecryptorProviderBuilder().build(password)
 JcaPEMKeyConverter converter = new JcaPEMKeyConverter().setProvider("BC")

 KeyPair kp
 if (object instanceof PEMEncryptedKeyPair) {
  kp = converter.getKeyPair(((PEMEncryptedKeyPair) object).decryptKeyPair(decProv))
 } else {
  kp = converter.getKeyPair((org.bouncycastle.openssl.PEMKeyPair) object)
 }
 kp
}

For security the password must be passed and processed as char [] instead of String object, since String is seen as one object and may be accessed before garbage collection.

The first challenge appear here that PEMParser (Thhe deprecated and removed class was PEMReader) class that is used in most of examples was not resolvable and accessible that was because of existence of old version in global grails lib through some dependency to garils-doc and itext (The dependencies can be checked by grails dependency-report command). So first solution was to exclude the bouncycastle lib in BuildConfig.groovy of my plugin by something like the following code:

dependencies {
 compile("com.lowagie:itext:2.0.8") {
  excludes "bouncycastle:bcprov-jdk14:138", "org.bouncycastle:bcprov-jdk14:1.38"
 }
}

It was OK for compiling source code in plugin project but same error of not founding some classes repeated on application project that use the plugin. So I added same to build config of application and it resolved. So now I can encrypt and decrypt strings by RSA keys in grails side.

I added syntax highlighter to blog template for this post.

First Taste of iOS, Swift and Xcode

Beyond the basics that can be found on many resources on the net with better quality, here is my founding about some details that take a little more time to found them:

• Show Document Outline, it is on left bottom of the standard editor of the story board. Here you can find constraint and views elements.
  
• Auto Syncing of View and its Controller, it can be enabled on top of assistant editor. On manual mode it appear as square with line in between that in Automatic mode is as 2 joined circle, like assistant editor icon.
  
• When using Ctrl + Drag to align horizontally or vertically, it align according to current position of element, so if it is inserted in the center at first, the spacing is zero, then it will go exactly in the center, but if not, it will go according to that space.
  
• Don't remove or rename an action method in the code, it should be updated in the Connections Inspector in utilities pane as well.
  

Verioning by Git Branchs

Version 0.1:

Create a branch (version)
git checkout -b 0.1.x

Do something / Commit / Push

git checkout master
git merge 0.1.x


Version 0.2:

git checkout -b 0.2.x

Do changes / Commit / Push

git checkout master
git merge 0.2.x

Groovy: Overloading [] and dot .

class Konf {

    // Initializing overloader
    static {

        // . read
        Konf.metaClass.getProperty = { String prop ->
            println "getProp $prop"

            this."${prop}"        

        }

        // . assignment
        Konf.metaClass.setProperty = { String prop, val ->
            println "setProp $prop $val"            

            this.metaClass."${prop}" = val
        }
    }

    // [] read
    def getAt( String key ) {
        println 'getAt'
        this."${key}"    

    }

    // [] set
    def putAt( String key, val ) {
        println "putAt $key $val" 

        this.metaClass."${key}" = val
    }

    def clear() {

    }
}

Restore MySQL Replication Failure

RESET SLAVE

START SLAVE UNTIL SQL_AFTER_MTS_GAPS

SET @@GLOBAL.slave_parallel_workers = 0

START SLAVE SQL_THREAD

Enable/Disable Windows Network Interface

Disable
netsh interface set interface name="Local Area Connection" admin=disabled

Enable
netsh interface set interface name="Local Area Connection" admin=Enabled

Reset
netsh interface set interface name="Local Area Connection" admin=disabled & netsh interface set interface name="Local Area Connection" admin=Enabled

Spring Roo 1.2.4, Spring Security 3.1

Customizing voters:

public class MyRightVoter extends RoleVoter implements AccessDecisionVoter
{
    public int vote( Authentication authentication, Object object,
Collection attributes )
    {}
}

Defining proper access decision manager and adding customized voter to it:
* To override http decision manager beans must be added to applicationContext-security.xml and for method based in controller must be added in webmvc-config.xml and add to it.

Adding new decision manager for method and http handling:

* aspectj mode is used when want to add security to our aspects like roo generated aspect (new aspect should be defined to annotate security to methods), this mode should be defined in applicationContext-security.xml, this global definition doesn't require to reference acess-decision-manage, as the main one that is in use, defined in webmvc-config.xml


* It is possible to customize authentication to return customized authority of users by implementing getAuthorities() method.

 * Don't forget to add the following item to spring/webmvc-config.xml to use annotation in spring controllers

Get Relative Path of Application Context

def s = ctx.grailsLinkGenerator

s.contextPath

s.serverBaseURL


The service is in the following package :
org.codehaus.groovy.grails.web.mapping.LinkGenerator

Use Services in Grails Shell/Console

def svc = ctx.getBean('serviceName')

def svc = ctx.serviceName

My Experience on Migrating a Database

I have a server that runs both application and database and all resources are utilized on peak time, so I migrated the database to 2 other new servers with more capacity.

It is about the whole story from OS installation to DB migration and the challenges on the way.

Use raid controller for mirroring 1st 2 disks for / and /boot.
The final layout of the root partitions is as:

sda 557.9G  0 disk
sda1 200M  0 part   /boot/efi
sda2 500M  0 part   /boot
sda3 557.2G  0 part
g_root-lv_root 424G  0 lvm    /
g_root-lv_swap 16G  0 lvm    [SWAP]
vg_root-lv_varlog 19.5G  0 lvm    /var/log
vg_root-lv_home 97.7G  0 lvm    /home

The /boot/efi may appear new that is because of using UEFI mode of BIOS boot agent.

Clearing old partitions of other disks after installing OS (RHEL):

For each disk that has partition
fdisk /dev/sd? -> p ->d -> w
or
dmsetup remove /dev/mapper/ddf???????????????????????


LVM:
pvcreate /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg
vgcreate -s 16M vg_data /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf
lvcreate --type raid6 -i 3 -L 1.5T -n lv_mysql vg_data
mkfs.ext4 /dev/mapper/vg_data-lv_mysql

Mounting Permanently:
/etc/fstab
/dev/mapper/vg_data-lv_mysql /data/mysql        ext4    defaults        1 2

SE Linux:
install semanage
yum install policycoreutils-python-2.0.83-19.47.el6.x86_64

Adding permission for new customized directory of mysql
semanage fcontext -a -t mysqld_db_t "/data/mysql(/.*)?"
restorecon -R -v /data/mysql

Firewall:
yum install system-config-firewall-1.2.27-7.1.el6.noarch

iptables -I INPUT 4 -m state --state NEW -m tcp -p tcp --dport 3306 -j ACCEPT
service iptables status
service iptables save

Note: -I INPUT 4 is putting this rule before the global rejection rule that is there by default, The status should be checked and ensure about the position of the rule, here 4.


MySQL:
Update /etc/my.cnf
mysql_install_db --defaults-file=/etc/my.cnf --user=mysql
service mysql start

Grant options for defining replication on both servers
mysql  -p -h 127.0.0.1 -P3306 -u root -e "GRANT ALL ON *.* TO root@'%' WITH GRANT OPTION;"

Install mysql utilities
yum install mysql-connector-python-2.1.1-1.el6.noarch.rpm
yum install mysql-utilities-commercial.noarch

mysqlreplicate --master=root@10.1.196.4:3306 --slave=root@10.1.196.12:3306 --rpl-user=rpl:sLavep@ss

Note: the --rpl-user values will be created automatically, no need to define that user.

Checking Replication Status:
mysqlrplcheck --master='root@10.1.196.4:3306' --slave='root@10.1.196.12:3306'

mysqlrplshow --master='root@10.1.196.4:3306' --discover-slaves-login='root'

Master Only:
mysql_secure_installation
mysql>
update mysql.user set user = 'dba', password=Password('123456'), host = '%' where user = 'root' and host = '::1';
SET sql_log_bin = 0;
delete from mysql.user where user = 'root';
SET sql_log_bin = 1;
flush privileges;

Note: SET sql_log_bin can stop/start replication of the current session.

SLAVE:
SET sql_log_bin = 0;
delete from mysql.user where user = 'root';
delete from mysql.user where host = 'hostname' and user = '';
SET sql_log_bin = 1;
flush privileges;

Checking replication status:
show slave status \G

Fixing replication failures:
Running empty transaction for failed transaction id (GTID)

SLAVE:
If there is error:
show slave status \G;
SET GTID_NEXT="2e82f8b8-cc26-11e4-bb69-0010e02446b8:3"
BEGIN; COMMIT;
SET GTID_NEXT="AUTOMATIC";
STOP SLAVE;
START SLAVE;
or
SET GLOBAL sql_slave_skip_counter = 1;

Redirecting old server processes to new DB server:
vi ~/.my.cnf
[client]
user=dba
password=123456
host=10.1.196.4

chmod og-rwx,u-w ~/.my.cnf

DB Migration:
Export:
mysqldump  --tab=./System/ System_db
mysqldump  --tab=./db2/ db2
mysqldump  --tab=./Dimension/ Dimension

Statistics Database:
mysqldump  --tab=./stat/ --no-data CS3_Statistics

Note: --no-data is used because I don't want to migrate all tables data in this phase

mysqldump  --tab=./stat/ --ignore-table=Statistics.ProcessStatistics --ignore-table=Statistics.Log --ignore-table=Statistics.Statistics_System  --ignore-table=Statistics.ServerIf --single-transaction Statistics &> /tmp/dump.log &

Note: --single-transaction is used to  prevent table lock.

Import:
mysqladmin create System_db
mysqladmin create db2
mysqladmin create Dimension
mysqladmin create Statistics

cat *.sql | mysql System_db
repeat for each db export

Note: In some cases that you have a view or foreign key you should ensure the order of sql and txt files to keep consistency. This order is not keeping by use *.sql or *.txt and you may need to list files manually.

mysqlimport --local Dimension Dimension/*.txt

MySQL Proxy:
stop old database server to free the listen port

Run mysql proxy to redirect connections to new server
./bin/mysql-proxy  --defaults-file=my.cnf

MySQL Connection Error Checking

Current Counts:
select * FROM performance_schema.host_cache;

SHOW GLOBAL STATUS LIKE 'Aborted_connects'

Clear Abort Connections:

mysql > Flush Hosts;

mysqladmin flush-hosts

Clear BIOS Raid Disks

Get Info:
dmsetup info
dmsetuo list

Clear Specific Device:
dmsetup remove /dev/mapper/ddf1_4c534920202020201000007910009263471147119133e6ca


Clear All Device (be cautious about operational and lvm disks as they are in the list):
dmsetup remove_all

Update the Boot Archive on a RAID 1 Volume

Check the current meta devices and mirror status before reboot
metastat -p
like d0 d10 d20

Go to fail safe mode:
ok boot -F failsafe or reboot -- "-F failsafe"

Mount primary submirror
mount /dev/dsk/c0t0d0s0 /a

Update vfstab
backup original cp /a/etc/vfstab /a/etc/vfstab.orig

Comment out / filesystem
#/dev/md/dsk/d0     /dev/md/rdsk/d10    /     ufs    1    no    -

Add sub-mirror disk as /
/dev/dsk/c0t0d0s0 /dev/rdsk/c0t0d0s0  /     ufs    1    no    -

Update system conf
Backup original cp /a/etc/system /a/etc/system.orig

Comment out rootdev
* Begin MDD root info (do not edit)
# rootdev:/pseudo/md@0:0,0,blk
* End MDD root info (do not edit)

Update boot archive
bootadm update-archive -R /a

Reboot
umount /a
shutdown -i 6

Detach faulty sub-mirror
metadetach d0 d20

Replace changed file
cp /etc/vfstab.orig /etc/vfstab
cp /etc/system.orig /etc/system

Reboot
shutdown -i 6

Attach sub-mirror
metattach d0 d20

Check resync
metastat

Meta Device in Single Mode

boot cdrom -s

mkdir /tmp/a

ls -lthr /dev/dsk

mount -o ro /dev/dsk/c0t5000CCA01535BEA8d0s0 /tmp/a

cp /tmp/a/kernel/drv/md.conf /kernel/drv/md.conf

umount /tmp/a

update_drv md

metastat /dev/md/dsk/d0

metastat

mount /dev/md/d0 /tmp/a

Restart Network Service

Solaris
svcadm restart physical

RedHat
service network restart

Ubunto
service networking restart

Python Enum

def enum(*sequential, **named):
    enums = dict(zip(sequential, range(len( sequential))), **named)
    reverse = dict((value, key) for key, value in enums.iteritems())
    keys = dict((key, value) for key, value in enums.iteritems())
    enums[ 'values' ] = reverse
    enums[ 'keys' ] = keys
    return type('Enum', (), enums)

ut = enum( M=1, S=2, T=4, V=8 )

ut.keys[ 'M' ] => 1

 ut.values[ 4 ] => 'T'

Manage Update/Remove of hasMany field

class ManagedTeam {
    String name
    static hasMany = [ users: ManagedUser ]
}

update method in controller:

//clear all users
managedTeamInstance.users = []

//add the selected ones back
params.users.each() {
    def ManagedUser user = ManagedUser.get( it )
    managedTeamInstance.users.add( user )
    log.debug( "in associateUser: added " + user )
};

//try to save the changes
if( !managedTeamInstance.save( flush: true ) )
{
    return error()
}
else{
    flash.message = "Successfully associated users"

}