Where we are

For the past few weeks we are trying to improve the functionalities of our system, so let's take a look at what have we done:

• We switched the exchange data format used for serializing from XML to JSON(GSON).
• Improve the CRUD functions, user can add more than one entry to database when he calls the POST method.
• We solve the problem of generating an unique identifier for an entry with Universally Unique Identifier.
• To assure that the database is consistent we added Transactions so each operation in the process of a transaction is guarantee to be atomic, which means that transactions are never partially applied. Either all of the operations in the transaction are applied, or none of them.

Now I will describe next tasks:

1. Authentication with Google accounts -  This kind of authentication is needed in order to use C2DM protocol (I will describe in a later post this feature) and to distinguish data between clients.
2. Delta Updates - In order to optimize the network traffic and save time I will focus on implementing Delta updates. This is an update that only requires the user to download the data that has changed, not the whole database. Any application ready for updating can be updated almost immediately due to this system. If, for example, a local database that is 100 megabytes is updated with a new amount of data that is 2 megabytes, the system will download only the 2 megabytes instead of 102 me gabytes.

In order to keep track of changes during development and testing phase we decided to create a repository. We have created a SVN based repository on Google Code.

Bigtable and The Skeleton of 3D for Android

      Bigtable is a distributed database system, designed to scale to a very large size(petabytes) across thousands of servers. It is owned by Google and used on some of their applications(more than sixty) such as Google Maps, Google Earth, Gmail and so on.

      It's closed source, although Google offers access to it as part of its Google App Engine. Since his deployment(late 2003) Bigtable has achived serveral goals: wide applicability, scalability, high performance, and high availability.

      Each table on this system is a sparse, distributed, multi-dimensional map where data is organized into three dimensions: rows, columns and timestamps.

(row:string, column:string, time:int64) → string

      In order to optimize the managing of a huge amount of data, the tables are split at row boundaries and stored as tablets. Each tablet hold contiguous rows and have between 100-200 MB distributed on several machines.

Each machine stores about 100 tablets(in GFS), this setup allowing good load balancing and fast recovery(if a system goes down, other machines take one tablet, so the load on each is fairly small).

       When sizes threaten to grow beyond a specified limit, the tablets are subject of three different type of compaction:

1. Minor Compaction - creates new SSTables - who has two goals: to reduce memory usage and reduce the amount of data that has to be read during recovery if the server dies.
2. Merging Compaction, periodically executed in the background, reads the contents of a few SSTables and writes out a new SSTable.
3. Major Compaction rewrites all SSTables into exactly one.

    More details about the implementation, data model and Google infrastructure on which Bigtable depends you can find on this lecture from University of Washington or on this paper.

       Regarding the small Rest API that I was about to develop, it has proved to be quite easy considering that I had some experience with Google App Engine, Jersey and Java. So I have created a small application on GAE and through Rest calls via Http I can Create(POST/PUT), Read(GET), Update(POST), Delete(DELETE) data in my table(Bigtable) on cloud. Also I created a simple application for Android who can do those operations as well.
      Now I will focus on describing the requirements of my future software implementation, but for that I have to do a research to find the best solutions  who fits.

The Idea

  

   With the fast advancement in information technology, database management systems are becoming more and more advanced. 

If at the beginning system designers and architects thought that a central control is better for database management, nowadays along with the relatively cheaper hardware, distributed database has become a better choice.

      The main purpose of the project is to allow users to reliably store and synchronize data between their mobile device and the cloud.  When there is not a network connection between the client and the cloud, the data is stored locally, and when the connection becomes available, the cloud database is updated accordingly. The database will be replicated to improve reliability, availability and fault-tolerance. The communication between clients and the cloud will be made through REST API and C2DM protocol. Android SDK, Google App Engine and Big Table are the technologies which will be used to develop this application.

       I think the picture below describe clearly the principle of the project. 

Google App Engine connected to Android Architecture

       This work will deal with the following problems: how to successfully integrate my system in an environment that allows replication and designing the algorithm for data selection.


       It will proceed along the following points:

• Analyze requirements for system storing data inside cloud (or environment allowing replication) allowing mobile devices to load relevant parts of the data. Focus only to database layer and interface.
• Study and describe basic principles of databases and distributed systems.
• Describe the algorithm used for data selection (range or context, etc.)
• Select one DB and document reasons for decision. 
• Design system architecture of system.
• Implement solution (DB side).

     This is a team project. I'll focus on the server side of the system, the client will be described and developed by my colleague Andreea Sandu.

    On the following days I'll describe the Big Table database provided by Google(GAE) and I'll try to develop a small REST Api in Java with the CRUD functionalities.

Until then, "Keep it Simple, Stupid"!