Requirements and Restrictions

   In this post I will try to describe the most important and natural requirements of my system.

Architectural Requirements

1. The system will be developed for Android Architecture using Google Technologies.
2. The persistence will be handled by a distributed database.
3. The database will be BigTable.
4. Database management will be provided by NoSQL.

Functional Requirements

1. The system will be designed for one application distributed on multiple mobile devices.
2. Transparent data (readable) will be supported by the system.
3. The system must be able to provide an internet connection to successfully synchronize the data between local database and cloud database.
4. The system will use a Wi-Fi connection when is available otherwise a GPRS standard - User can choose what type of connection will be used primarily, as default Wi-Fi will be set up. Also user application will be able to force this relation: in case only GPRS is available the application will load only relevant data.
5. Offline work will be supported by the system - The accuracy of the application will not be affected by the fact that at a time there is no available network connection, Content Provider will store data locally(cache approach) and synchronize it with the cloud when an internet connection becomes available.
6. Cloud to Device Messaging(C2DM) protocol will be used to tell the application that on the server side is new data, so that the application can fetch it. The C2DM service handles all aspects of queueing of messages and delivery to the target application running on the target device.
7. JSON(GSON) will be the data exchange format used for serializing and transmitting structured data over a network connection.
8. To ensure consistency the system will be able to replicate the data on several storage devices.
9. Timestamps will be used to solve conflicts during resynchronization. We will use the "last update wins" approach because it's the most simple and natural.
10. The communication between entities(server and client) will be ensured through REST architecture using Create/Read/Update/Delete functions.
11. The database on the server side will be able to maintain and distinguish pieces of data from multiple clients.
12. Optimize the network bandwidth and save time making Delta updates. This require the user to download  the data that has changed, not the whole database.

Non-Functional Requirements

1. The programming language used for developing the system will be Java.

     Beside those requirements, at the beginning I will add some restrictions just to be sure that in the end I will have a functional software. If the development of the project will go well, I will remove step by step one of them and I will try to solve it. The complexity of my system will increase with every removed restriction.

Restrictions

1. Network connection is available at any time.
2. One device will be used to avoid conflicts in synchronizing the cloud and local database.This implies that a network synchronization will take place before continuing working.
3. The client database will fit in the cellphone memory.
4. Every application has its own database.
5. Database from the server side (cloud) will be accessible only from Android devices.
6. Changes on the client side will result with an update of the entire database on the cloud.

     The software development methodology that I will try to use is Agile. Those methods have proven their effectiveness and are transforming the software industry everyday. The main goal is to provide a functional software. Tasks will be divided into small increment parts and functional software is developed in short iterations("timeboxes"). A good feature of this method is that encourages rapid and flexible response to change.

     

      It seems this method uses my design principle, Keep it simple, stupid!, we will see if is true.

Bigtable and The Skeleton of 3D for Android

      Bigtable is a distributed database system, designed to scale to a very large size(petabytes) across thousands of servers. It is owned by Google and used on some of their applications(more than sixty) such as Google Maps, Google Earth, Gmail and so on.

      It's closed source, although Google offers access to it as part of its Google App Engine. Since his deployment(late 2003) Bigtable has achived serveral goals: wide applicability, scalability, high performance, and high availability.

      Each table on this system is a sparse, distributed, multi-dimensional map where data is organized into three dimensions: rows, columns and timestamps.

(row:string, column:string, time:int64) → string

      In order to optimize the managing of a huge amount of data, the tables are split at row boundaries and stored as tablets. Each tablet hold contiguous rows and have between 100-200 MB distributed on several machines.

Each machine stores about 100 tablets(in GFS), this setup allowing good load balancing and fast recovery(if a system goes down, other machines take one tablet, so the load on each is fairly small).

       When sizes threaten to grow beyond a specified limit, the tablets are subject of three different type of compaction:

1. Minor Compaction - creates new SSTables - who has two goals: to reduce memory usage and reduce the amount of data that has to be read during recovery if the server dies.
2. Merging Compaction, periodically executed in the background, reads the contents of a few SSTables and writes out a new SSTable.
3. Major Compaction rewrites all SSTables into exactly one.

    More details about the implementation, data model and Google infrastructure on which Bigtable depends you can find on this lecture from University of Washington or on this paper.

       Regarding the small Rest API that I was about to develop, it has proved to be quite easy considering that I had some experience with Google App Engine, Jersey and Java. So I have created a small application on GAE and through Rest calls via Http I can Create(POST/PUT), Read(GET), Update(POST), Delete(DELETE) data in my table(Bigtable) on cloud. Also I created a simple application for Android who can do those operations as well.
      Now I will focus on describing the requirements of my future software implementation, but for that I have to do a research to find the best solutions  who fits.

The Idea

  

   With the fast advancement in information technology, database management systems are becoming more and more advanced. 

If at the beginning system designers and architects thought that a central control is better for database management, nowadays along with the relatively cheaper hardware, distributed database has become a better choice.

      The main purpose of the project is to allow users to reliably store and synchronize data between their mobile device and the cloud.  When there is not a network connection between the client and the cloud, the data is stored locally, and when the connection becomes available, the cloud database is updated accordingly. The database will be replicated to improve reliability, availability and fault-tolerance. The communication between clients and the cloud will be made through REST API and C2DM protocol. Android SDK, Google App Engine and Big Table are the technologies which will be used to develop this application.

       I think the picture below describe clearly the principle of the project. 

Google App Engine connected to Android Architecture

       This work will deal with the following problems: how to successfully integrate my system in an environment that allows replication and designing the algorithm for data selection.


       It will proceed along the following points:

• Analyze requirements for system storing data inside cloud (or environment allowing replication) allowing mobile devices to load relevant parts of the data. Focus only to database layer and interface.
• Study and describe basic principles of databases and distributed systems.
• Describe the algorithm used for data selection (range or context, etc.)
• Select one DB and document reasons for decision. 
• Design system architecture of system.
• Implement solution (DB side).

     This is a team project. I'll focus on the server side of the system, the client will be described and developed by my colleague Andreea Sandu.

    On the following days I'll describe the Big Table database provided by Google(GAE) and I'll try to develop a small REST Api in Java with the CRUD functionalities.

Until then, "Keep it Simple, Stupid"!

Hello World!

Who am I?


       As I said in my short description, I am an exchange student at ČVUT, Prague. I am in my final year of a Computer Science Bachelor degree. I'm intrested in learning and excelling in new technologies and use my education and experience to accomplish my goals. 

Currently I’m focused on cloud computing, distributed systems and web development based on Android platform. I have experience in C, Java, SQL and Networking.

What is this blog about?


       During my thesis preparation I would like to share information about my progress and to describe step by step each milestone. Also I want to encourage everyone to give feedback and suggestions.

Acknowledgements

I would like to thank to my mentors Ing. Jan Sedivy, CSc. and Ph.D. Tomáš Bařina for their trust, time they will spend for regular consultations and willingness to share their rich experience.

I will describe in a further post the main idea of my project and the content of it.

P.S. It's not so simple as I thought to write a post, those few rows took me more than one hour. I will get used to it :).


"Keep It Simple, Stupid!"