Safe DB Schema changes Using JOOQ for rolling deployment

In order to explain this concept, we will use auth-service as an example. The same concept applies for any other service and any other number of service instances.

Auth-service architecture for multiple instances

In this architecture,

  1. we are using one instance of proxy-service, four instance of application service (auth-service instances in this case)
  2. Single instance of DB.
  3. As a result, if any auth-instance updates database, the changes are visible to all auth-instances.
  4. This is the reason we apply db-changelog using only one auth-service instance.


Downtime deployment

  1. In downtown deployment, we shutdown all auth-service for a timeline during which time, no request is served.
  2. During this downtime, we apply all the changes to database and start auth-service instance.
  3. Again all instance share same database status.

Rolling / no-downtime deployment

  1. As we are moving toward no-downtime release, we cannot stop all instance of auth-service to deploy changes.
  2. We need to deploy new changes to one auth-instance at a time. Hence, the name rolling deployment.

This is how rolling deployment works:

  1. We stop the auth-instance where we like to install changes
  2. Before stopping the auth-instance (say, auth-service instance1), udpate auth-proxy that it would be offline and wont handle any request.
  3. Rundeck installs updates for auth-service instance1 , do changes with chef-client and so on.
  4. Auth-instance1 become active and update auth-proxy that it is ready.
  5. 1 to 4 happens to all instance of auth-service.


Database  schema changes

Lets now assume we are adding a new column (object_id) in role_membership table where existing columns are user_id, role_id and tenant_id.


user_id role_id tenant_id object_id



We have following choices:

  1. Add object_id as the last column of the table (Recommended)
  2. Add object_id in any other column


Note that, our new deployment is about rolling these schema changes across all auth-service instances.


JOOQ time


As we are using JOOQ as the object relational model (intermediate layer between DB and application), for our desired schema changes, we first generate right JOOQ changes. Our auth-changes contains this changes for JOOQ.

How JOOQ maps column in JOOQ record


JOOQ uses integer index starting from index Zero to point to a column in a table in database.


index used by JOOQ 0 1 2

Table 1: JOOQ state of old auth-service instance


our JOOQ code without object_id is containing a configuration like the table above.

DB-inconsistency problem for rolling deployment for newly added column position in middle


Now assume a scenario, where we have added new column object_id in the middle of the role_membership table. At this time, we have JOOQ mapping to the table as shown below.


index used by JOOQ 0 1 2 3

Table 2: JOOQ state of  incorrectly updated auth-service instance 



Now that

  1. auth-instance running latest code will point object_id by index 2
  2. auth-instance will old changes will point tenant_id by index 2


We would have not got to this problem if we have added object_id column at the end of the table as shown below. See that in this case both old code and new code of JOOQ points to same column by same index (comapre table 3 and table 1)

index used by JOOQ 0 1 2 3

                      Table 3: JOOQ state of  correctly updated auth-service instance 



Take away

As we are moving toward no-downtime/rolling deployment, whenever we do DB schema changes to add a new column in an existing table, we need to add the column at the very end of table.

Categories: Uncategorized

Python: string.split() vs re.split()

While both string and regular expression in python has split() method, string.split() can split using only one separator.

def split_with_string():
    s = "how-r-you."
    assert s.split("-") == ['how', 'r', 'you.'] #True

On the other re.split() can split using regular expression. When you are using, you can build any complex separator or combine multiple separators.

def split_with_re():
    import re
    s = "how-r-you."
    assert re.split(r'[-|.]') == ['how', 'r', 'you', ''] #True

Categories: Uncategorized

Python ‘in’ operator with Iterator

I found use of in operator with iterator very useful. It would help write functional style code more easily.

list = [1,2,3,4]
list = iter(list)
assert 1 in list == True 

Explanation: iterator is matched until value 1 is matched, which is matched in the first place (index 0). would return 2 in this case.

list = [1,2,3,4]
list = iter(list)
assert 5 in list == False # Throws exception

Explanation: 5 does not occur in the list. The list is all traversed to find occurence of 5. would raise StopIteration exception

Now lets see how to use ‘in’ operator with iterator to do something useful. Following function is written to check if the second argument is a subsequence of first argument.

def IS_word_subSequence_of_s(s,word):
 startIdx = 0
  for char in word:
    startIdx = s.find(char, startIdx)
    if startIdx == -1:
     return False
     startIdx += 1
 return True

Instead I found this code snippet [1] that very nicely use functional style to do the same thing.

def isSubsequence(s,x):
        it = iter(s)
        return all(c in it for c in x)


Categories: Uncategorized

Preserving states across function call in Python

Usually functions in Python is not state preserving. That means functions do not save values between calls. But you can make it happen in the following way


# this function remember total from last calls
def summation(value): = getattr(summation,"total",0) + value

When execute:

summation(1) == 1 # true

summation(2) == 3 # true

summation(3) == 6 # true


Saving states across function call might be useful. For example, here I am trying to accumulate sum over a list of numbers.

# clear last total value = 0

[summation(x) for x in [1,2,3]] == [1,3,6] # true


Categories: python, Uncategorized

OpenStack Austin summit in pictures


Heading to OpenStack summit




@ Austin Convention Center entrance.




@ Austin Convention Center entrance.




OpenStack crowd.




Brand yourself with OpenStack




Under OpenStack umbrella. 




Youngest OpenStack enthusiast































Break out sessions.




@ Break.








Marketplace Theater.




ORACLE @ marketplace




ubuntu @ marketplace




vmware @marketplace




Mirantis @marketplace





Categories: openstack, Uncategorized

Hard link vs soft link

The following figure explains hardlink, softlink and    their association with inode. The figure is adopted from [1]









Categories: Uncategorized, Unix

su vs sudo command

The command su (short for substitute user) is used to switch user in UNIX environment. When we issue su command  with/without a username, it switches to root or otherwise user mentioned.


On the other hand, sudo command run a single command with root privilege. By leveraging sudo, the system discourages to switch to root user when it is not absolutely necessary.


We can su with -c  to achieve the same as sudo command.



#Prosunjit Biswas#


Categories: Uncategorized, Unix