Try You will like it.

So you tried docker a little bit, decided that it is better than whatever you had before and want to jump on it. But if you have an existing mature continues integration pipeline, you may hesitate. Because right now after you commit your code change, the rest is taken care of by your CI pipeline automatically, build, test, deploy, etc. Only when their is a problem you will be notified. However if you use docker, you find yourself need to manually type a lot of commands or start to write a lot of scripts with no pretty UI anymore. You must wonder if someone has already did this for you and the answer is Yes.

Introducing , the one and only and best end to end solution for docker based CI pipeline. Disclaimer, I am not affiliated with them at all. I simply love their product. I switched everything I own from Heroku to here even though Heroku is free but you need to bring your own server for . (Heroku is a great product for its time, I would have totally wrote a blog like this couple years ago if I was blogging)

I have been using for a while. I enjoyed it all the time. But only recently I finally decided to recommend them to the world because they just solved a big problem I have. Now I think you can ignore their “Beta” flag and use them for production. (Personal opinion, not responsible for any decision you make) The problem I had was that Docker Hub sucks. I had several private repos their and I have encountered severl times that the build got backed up or completely stuck there results in the change I want to get out cannot. I don’t even make changes that frequent so I bet their actual SLA is even worse than what I experienced. It is understandable that they encounter this problem because everyone in the world is asking them to build their docker image all the time. Also they are the inventor or docker, they are system guys, probably not very experienced with service. solved the problem by leverage the machine of each person has, now the machine/node that I eventually will run the docker is also in charge of building the docker image. They also use the machine to run test of the docker and even though I’m not using that feature right now, believe me I will soon. And I have researched other solutions: Travis CI, Circle CI,, some docker solutions on top of Jenkins, believe me, none of them is doing it right.

Give a try and let me know your experience down in the comment.

REST on top of SQL. Someone please build it.

Existing similar things: (from worst to best)

Not looking good. Seems to charge a lot of money and not open.

Looks very crappy and not active. All XML

Looks a little less crappy. Does have a github . And looks does have some activity.

Very barebone. Things are configured with XML. But it’s Java and can be extended. The docs and stuff makes sense. Can be used as a Java Library, but if used like that, why not use a proper ORM like Hibernate?

Looks easy to use. Written in python and heavily influenced by Django clearly. Not active for a year, but the automatically inspect db using reflection is a good idea.

Django Tastypie and Django REST framework. These two looks good but then you have to buy in Python Django. And still a lot of things to setup and code to write.

Looks good and professional. Concern about how open it is. It says Apache 2.0 Licence though, so maybe good. Very active Written in PHP. Seems to have much broader range, stuff like client side library etc. Support SQL, NOSQL, etc and more importantly, seems at the same time. Even have a Docker image ready.

Seems to be the best choice so far.

My requirements:

  1. REST to SQL CRUD operations.
  2. REST to SQL LFSP (I invented this List Filter Sort Pagination) operations
  3. Cross table joins. Nested return. This maybe optional in an AJAX world. But may be necessary for complicated logics. However, when you consider if this supports multiple data sources, you cannot avoid in memory join, so maybe an in memory join support is more useful. Maybe you need to query one data source and then query another based on previous result. IN clause maybe a problem here since MySQL is not very good at it and Oracle even have a 1000 limit.
  4. API version controls that supports multiple versions.
    1. Maybe you simply need multiple sets of things completely (including different databases for different versions. Let’s face it, different versions probably require different db schema) and you have another service layer on top and when a request comes in, you call call one version easily but if it is a write request, you need to convert the request to make sense for other versions and make additional requests. Still this poses a big problem for data out of sync.
    2. Maybe you have multiple tables for different versions but the transaction will update all of them. Less problematic because it is in a transaction?
  5. Automatically generate documentations.
  6. An admin UI for modifying the data
  7. CLI for the following admin stuff:
    1. Configure database connection results in configuration file. Commit to git? Maybe only the connection and configuration but not the credentials.
    2. Create new edit existing table/resource results in schema migration and data migration files. Commit to git? Or should this be in database? Or should the schema migration be in git and data migration in database.
    3. Automatically have createdAt and updatedAt ?
    4. Optionally have complete history on some table? I don’t mean SQL history. That is in a big SQL log already. But history with who modified what. Very useful for audit. How does this work when schema changes?
    5. Use existing db through reflection. But this may not be necessary. Or this could be the killer feature.
  8. Maybe a JSON REST API and admin UI for these admin work above. But keep in mind that when there is an API and UI, how can we commit to git for the changes? Maybe should learn from the versioned wordpress.
  9. ACL Access Control Level. This may exists in a completely different layer on top of what we have above. We are not going to create a mysql user for each user of the system to leverage database’s access control, that will create a lot of connections to the database. So although ACL data is saved in database but it has to be the service that make sense of it and grant or deny access.
  10. Customize business logic.
  11. Support NOSQL and other types of datasource.
  12. Support multiple and multiple types of datasource together. I definitely feels better if this is another layer on top.

Docker: compare to virtual machine part 3

In my previous post, I mentioned that Microsoft is embracing docker, partnering with them, totally jumped in to this Docker wagon. As a former Microsoft employee, that was really kind surprise to me. I feel Microsoft has become really a different company with a new CEO. Also, to correct 2 of my friends working in Microsoft: NO, Docker does not support native Windows yet as of 5/30/2015. It will though, see here.

In this post, I’m going to talk about another company that really surprised me in this Docker world – CenturyLink. Yeah, you heard me, the CenturyLink cable company that did a lot of cheesy ads on $19.99 high speed internet for 5 years which is actually DSL and bundles with phone service. I used their DSL couple years ago. The experience was in one word: nah.

But, I have no clue what got into them or maybe their Innovation Lab is a completely separated entity, they seems to be a big player in this new Docker trend. I read quite a few good articles from them about Docker and watched several interviews the conduct with key person from Docker Inc. And more related to this post, they have this amazing tool that can really help you understand what a docker image looks like. Let’s load the sample images and see what we can learn.

Sample docker image layers

Sample docker image layers

Now, image these are cakes that are upside down. Each cake have different layers but they may have the same base. And the baker can bake one huge base and then put different stuff on top of them to become different cake. Some cakes can be very similar and share more than one base but several layers. In the picture above, ruby, python and node have the first 5 layers exactly the same. Only start to be different on the 6th layer.

Unlike a cake, and unlike virtual machine, the base layers is re-used in docker and it not reused by copy paste but reused as in there is only one copy of that layer needed. So say if you have python already on your machine with all the layers python need, and now you want to get ruby, it only need to pull down 5 layers instead of 10 since you already got the first 5 layers of ruby. Compare this to a virtual machine. Say you have a virtual machine running Windows, and then on top of it Microsoft Word, now you want a virtual machine running Windows and Microsoft Excel, you have to create a virtual machine image that have a completely separated Windows instance.

Docker: compare to virtual machine part 2

Continue from part 1, Now let’s talk about Docker. First let’s talk about a limitation. It is on Linux only. So you are out of luck if your host OS is Windows, or your services need to run on Windows, for example ASP.NET on IIS server or Microsoft SQL Server 2014. I have to point out that actually Microsoft is partnering with Docker actively. The post-Ballmer Microsoft actually is like a different company. However Docker targeted Linux only, so all they can do is to create a good Linux virtual machine on top of windows like this, which in my opinion definitely has drawbacks. However it is the same situation for Mac. I have to run Boot2Docker to create a VM to develop Docker on Mac too.

But Linux is dominant in server world anyway. Focusing only on Linux also have some benefits such as easy to implement and support, better performance, etc. This reminded me of Java Virtual Machine, which can truly ran on a lot of different operating systems, and got quite some criticism for it’s performance. (It can only run Java program of course but that is not it’s fault.)



Come back to this picture again, Docker runs directly on top of the host OS just the hypervisor (I wonder if one day we will have firmware Docker or even hardware Docker just like hypervisor), but on top of Docker there is no guest OS. The application directly talks to Docker. This is a huge advantage for simplicity of configuration and performance. You do not need to worry about different types of guest OS, different versions of guest OS, the security issue of the guest OS, the maintain, configuration, monitoring, etc. And also when you start up a Docker container or turn off a Docker container, it is much much faster than start and turn off a virtual machine which need to start the guest OS and stop the guest OS. According to this slide, it takes <50ms to start/stop Docker container but 35-50s to start and stop virtual machines. Do not have a guest OS is not the only reason that Docker can do this so fast. Re-use of the image is another big part which we will talk in next blog post.

(To be continued.)

Docker: compare to virtual machine part 1

They are very different of course. Docker is a type of Linux container (operating-system level software container specific to Linux). Pictures on this page is very helpful on understanding it. Now I’m going to try to explain it to you in my own words and examples hopefully you can relate to.



There are many computers operating systems. In personal computer the top 3 are: Windows, OSX and Linux. Each with many different version and flavors. Linux as a personal computer OS is mostly among developers and after Ubuntu provided a half decent GUI. Most people, when buy a personal computer, got either a PC or a Mac and it comes with Windows and OSX correspondingly. BTW, these 2 camps don’t like each other. Here is the proof.

However, occasionally because of necessity (say you need to use a software only available on the other OS) or boredom, you may need/want to be able to use both Windows and OSX. But you only have one machine, either a PC or a Mac. One way to solve this is to multi-boot. Both PC and Mac support this and you can install both Windows and OSX on a PC or Mac. It is just very hard to do. And you cannot use both operating systems at the same time.

A much easier solution would be use a virtual machine. You need to use a virtual machine software such as Virtual Box, VMWare, etc. There is a long list of such software. Using them, you can create multiple virtual machines easily and relatively faster compare to multi-boot. And you can (actually have to) use them together with the original OS on your Mac or PC. You will notice that each virtual machine you create will be saved by the virtual machine software as a big file. You can easily backup the file, move the file, even copy paste the file to create more identical virtual machines. You will also notice that the virtual machines are slow. Sometimes unbearable depends on how good is your computer. Well have a look at the picture about virtual machines on this page again. Of course it is slow. An operation need to go into the software, then the guest OS, then the virtual machine software (hypervisor in the picture, I’ll explain later), then the host OS, and then the hardware. Especially the guest OS, who has no idea that it is actually running in a virtual machine could really slow things down and it is also why the virtual machine file is very big.

Now in the server world, Linux is dominant, Windows has a meaningful percentage and I’ve never heard of Mac being used as a server though I’m sure it can. And the data center machines are huge monsters compare to personal computers. Take a look at this PowerEdge R920 from Dell. All the cloud service providers such as Amazon EC2, Microsoft Azure, etc all have quite a few data centers with a lot of powerful machines in them. And they rent them out to a lot of other companies to run their services or applications. A service may only need a tiny fraction of one server’s power, and it may suddenly need a lot more (say a e-commerce website during Christmas) and then go back to need very little. The solution is that these huge servers will start and stop virtual machines on demand. They use SSD to do disk I/O faster to help start the virtual machine faster and give it better performance. And also a lot of effort are put into make good hypervisors so the performance is good. Hypervisor is a broader concept including virtual machine software you use on your personal computer but also include specialized software and sometimes are firmware or even hardware that are used in data center with high performance.

(to be continued)

Docker: How did I find it.

I don’t really remember the exact moment I saw Docker for the first time. It’s roughly September 2014 and I definitely did not dig into it for another several months, not love at first sight. Later, my team moved production code from one data center to another in November 2014, I talked about it in one of my previous post. The process was quite chaotic and tedious. Also, we moved from CentOS to Ubuntu at the same time. During that process, I learnt a lot IT stuff that I did not have much previous exposure to as a developer: chef, puppet, salt, etc.
After took a vacation in the holiday season, early 2015 (also around when I dropped the ball on this blog) I started researching (i.e Google) for better technology/system for configure production environment and easy deployment. That’s when docker came back to me and I took a deeper look. The first time I saw the blue whale icon on I though of the famous fail whale from twitter. Their interactive tutorial is pretty cool and fun. I went through it so fast the first time that I missed things and had to do it twice. But I’m totally OK with it, that’s how fun it is. I highly suggest you go do it now if you are a developer or comfortable using a command line window.

docker whale


Twitter fail whale

Twitter fail whale

Docker: I’m pretty sure it’s the future.

Docker is hot and getting hotter. Here is the Google Trend screen shot as of today, May 16th, 2015.

Docker Google Trends result May 16th, 2015 Screen shot

Docker Google Trends result May 16th, 2015

I don’t know when are you reading this post, so here is the real time Google Trend embedded. I bet it is going to be even better than the screen shot.

There are a few good reasons why it is so popular. There are tons of articles on the internet about benefit of docker. I’m not an expert on docker yet, just a normal user so I will only talk about my experience using it and the benefits from my perspective as a developer in my next couple blog posts.