DockerCon 2016 summary

I attended DockerCon 2016 during June 20th and 21st. Feels pretty lucky that some convention center in sillicon valley need renovation so DockerCon choose to be held almost right next door to me. Today I finally finished watching videos of the talks that I did not go to here. And I have been playing with the Mac Beta version of Docker for the last month, so it’s time for this brief summary.

  1. Docker community is growing fast! Everyone is onboard now. Those are not will be irrelevant soon.
  2. Docker Inc, is producing tons of stuff. So many new features in the Beta. How did they do it? Well some purchases I’m sure like Tutum which I really like.
  3. DevOps is really interesting but if you dive deep it is hard. Respect to all DevOps!
  4. I feel know so little about my own trade (service and application development), after watching this video https://www.youtube.com/watch?v=pD0rEtEEwck

How many creative projects should a person work on simultaneously?

This question was discussed in one of the podcasts I listen to called Question of the Day. Their answer is 5-10. I think that is very true. I am currently stuck with one of the projects I’m working on. The project is CraigsMenu. Also checkout the blog there if you can. I have been trying to push myself to work on it for the past 2 days but for whatever reason, I simply cannot. I feel like I’m in a video game and this particular magic spell has been used too much and now requires a very long cool down period. I’m going to switch to something else and I hope it can solve the problem. Now it’s time for me to pick another creative project. Already feeling excited!

Amazing story about Leicester City F.C.

Summary of the story: Leicester City F.C. is a small soccer club in British Premier League. They only got into the league after 13-14 season. In 14-15 season, they almost got demoted. They were at the bottom after 29 of 38 games. But against all odds, they started their crazy come back, won 7 of the rest 9 games and survived. The unbelievable part is in 15-16 season. Because their performance was not that great in 14-15 season, their odds of winning the champion, according to some sports gambling company, was 5000:1 . They also got a new manager (head coach) that was not considered a good choice by most. They do not have tons of money to buy any super stars. Their main striker Jamie Vardy was still playing non-league soccer in 2012. However, against all odds, including the 5000:1 by the gambling company, they won the champion leading second 10 points. Jamie Vardy also broke the record of score most in consecutive games.

I’m not a huge soccer fan so I only learn this story today. It is really amazing. It’s a classic underdog story. I really feel inspired by it. Some high school teacher even put sign in class room to encourage students saying “If Leicester City can win the champion, you can do anything.”

Go Fox!

Falsifiability

Recently I’m bothered by insomnia. There is no one else but me to blame. However one night I encountered a video about Falsifiability by teachphilosophy on Youtube. I made some comments and to my surprise techphilosophy replied nicely with a lot of constructive discussions. This kind of experience is very rare on internet. I exchanged more comments with him/her and clearly he/she has deep philosophy background and my thoughts are much less sophisticated and scattered. “less sophisticated” in my opinion may or may not be a bad thing but “scattered” definitely is. So I hope I can do a better job with this blog post than the comments I have on Youtube.

First, if you are not familiar with Falsifiability, please read wiki. Now, let’s talk about all the examples in techphilosophy’s video.

1. There is a planet between Mercury and Earth. This is a pretty straight forward falsifiable statement, unless IAU went nuts again and keep changing the definition of “planet” then this statement is not well defined. But I want to talk a little about how do we know that the statement is true. We know planet Venus is there between Mercury and Earth, but how can you say “you know”? I’m pretty sure you have not been to Venus and feel it with your hand. You probably have seen it in the sky but how do you know it is a real planet but not a man made satellite? And how do you know it is between Mercury and Earth? Well, we have to use the Occam’s razor. You can safely assume that all astronomist in the world did not create a big Venus hoax for no reason than fool you.

2. All swans are white. Classic example of something falsifiable and got falsified. I don’t have much to say about it.

3. Nonspatial/Nontemporal Fairies live inside my nose.

4. techphilosophy skipped it so no idea what did he put there. But I’m guessing it may be “God exists” and he does not want to get into trouble.

5. I am currently conscious. This is the one where the most discussions happened between me and techphilosophy. His/her opinion is, if I may summarize, this is not falsifiable but very valuable and useful knowledge. I think it may or may not be falsifiable depends on how you define certain words, and when you make it not falsifiable, it is a topic as valuable and as ridiculous as #3. In my opinion, strictly non falsifiable statements are all equivalent. Because of historical reasons, philosophers still likes to cling on to this topic but with advance in neuroscience, they are clearly loosing the grip. So let’s start with definition of conscious, here are some that Google shows:

  • aware of and responding to one’s surroundings; awake.
  • having knowledge of something; aware.
  • painfully aware of; sensitive to.
  • concerned with or worried about a particular matter.
  • (of an action or feeling) deliberate and intentional.

From dictionary.com

  • aware of one’s own existence, sensations, thoughts, surroundings,etc.
  • fully aware of or sensitive to something (often followed by of):
  • having the mental faculties fully active:
  • known to oneself; felt:
  • aware of what one is doing:
  • aware of oneself; self-conscious.
  • deliberate; intentional:

Let’s start with a simple one: deliberate;intentional. Sounds like this will get used a lot in court, right? And who decide if accused was or was not deliberate/intentional? Not himself/herself, it will be judge or jury based on their opinions hopefully based on facts. So for this definition only what other people thinks matters. Remember this because it will be useful later when we talk about other definitions that seems does not involve others but only about oneself.

Next, several of the definitions are about aware of things, surroundings. This is another thing that can be hotly debated in a court since it can also impact the outcome a lot. But again, debated by other people, only what others thinks matters. It probably shows up more in hospital though. Still decided or tested by other people, doctors, nurses, etc.

Lastly, let’s discuss the one that does not involve other people “aware of one’s own existence”. Put that in the statement, it becomes: “I am currently aware of my own existence.” The idea is that when a person makes this claim in his/her own head, he/she is certainly not sleeping or unconscious and this becomes a self fulfilling statement and is always true. And because it does not involve others, it is not falsifiable. Well, with the advance in neuroscience, we can easily tell if a person is awake or dreaming and with advance in machine learning, we can even guess what a person is seeing (link). But even ignore that, let’s say no one is attaching any EEG machine to your shaved head, no one is even around you to observe you, no camera is recording what are you doing for anyone to see in the future, let’s try to make the environment allow the statement that “you are currently conscious” truly unfalsifiable. Then you can think to yourself “I am currently conscious”. But are you?? It has become very similar to the question: “If a tree falls in a forest and no one is around to hear it, does it make a sound?”. Except you say that you are the tree and you are there to hear it. Well, if you perceive that you are conscious as if the tree was able to make a record of the sound when it fell, then the statement becomes falsifiable. It is possible that in the future we will have a machine that can read your memory. So you have to forget your experience to make it truly non falsifiable.

In conclusion, I am not exactly sure what am I trying to say. I think what I’m suggesting is to apply falsifiability strictly to a topic before we decide to engage. I believe it will save a lot of meaningless discussions and a lot of time.

 

 

Try tutum.co. You will like it.

So you tried docker a little bit, decided that it is better than whatever you had before and want to jump on it. But if you have an existing mature continues integration pipeline, you may hesitate. Because right now after you commit your code change, the rest is taken care of by your CI pipeline automatically, build, test, deploy, etc. Only when their is a problem you will be notified. However if you use docker, you find yourself need to manually type a lot of commands or start to write a lot of scripts with no pretty UI anymore. You must wonder if someone has already did this for you and the answer is Yes.

Introducing tutum.co , the one and only and best end to end solution for docker based CI pipeline. Disclaimer, I am not affiliated with them at all. I simply love their product. I switched everything I own from Heroku to here even though Heroku is free but you need to bring your own server for tutum.co . (Heroku is a great product for its time, I would have totally wrote a blog like this couple years ago if I was blogging)

I have been using tutm.co for a while. I enjoyed it all the time. But only recently I finally decided to recommend them to the world because they just solved a big problem I have. Now I think you can ignore their “Beta” flag and use them for production. (Personal opinion, not responsible for any decision you make) The problem I had was that Docker Hub sucks. I had several private repos their and I have encountered severl times that the build got backed up or completely stuck there results in the change I want to get out cannot. I don’t even make changes that frequent so I bet their actual SLA is even worse than what I experienced. It is understandable that they encounter this problem because everyone in the world is asking them to build their docker image all the time. Also they are the inventor or docker, they are system guys, probably not very experienced with service.

tutum.co solved the problem by leverage the machine of each person has, now the machine/node that I eventually will run the docker is also in charge of building the docker image. They also use the machine to run test of the docker and even though I’m not using that feature right now, believe me I will soon. And I have researched other solutions: Travis CI, Circle CI, Drone.io, some docker solutions on top of Jenkins, believe me, none of them is doing it right.

Give tutum.co a try and let me know your experience down in the comment.

REST on top of SQL. Someone please build it.

Existing similar things: (from worst to best)

http://www.espressologic.com/features-rest-api/

Not looking good. Seems to charge a lot of money and not open.

 

http://sqlrest.sourceforge.net/

Looks very crappy and not active. All XML

 

http://restsql.org/doc/Concepts.html

Looks a little less crappy. Does have a github https://github.com/restsql/restsql . And looks does have some activity.

Very barebone. Things are configured with XML. But it’s Java and can be extended. The docs and stuff makes sense. Can be used as a Java Library, but if used like that, why not use a proper ORM like Hibernate?

 

https://github.com/jeffknupp/sandman

Looks easy to use. Written in python and heavily influenced by Django clearly. Not active for a year, but the automatically inspect db using reflection is a good idea.

 

https://github.com/django-tastypie/django-tastypie

http://www.django-rest-framework.org/

Django Tastypie and Django REST framework. These two looks good but then you have to buy in Python Django. And still a lot of things to setup and code to write.

 

http://www.dreamfactory.com/

Looks good and professional. Concern about how open it is. It says Apache 2.0 Licence though, so maybe good. Very active https://github.com/dreamfactorysoftware/dsp-core Written in PHP. Seems to have much broader range, stuff like client side library etc. Support SQL, NOSQL, etc and more importantly, seems at the same time. Even have a Docker image ready.

Seems to be the best choice so far.

My requirements:

  1. REST to SQL CRUD operations.
  2. REST to SQL LFSP (I invented this List Filter Sort Pagination) operations
  3. Cross table joins. Nested return. This maybe optional in an AJAX world. But may be necessary for complicated logics. However, when you consider if this supports multiple data sources, you cannot avoid in memory join, so maybe an in memory join support is more useful. Maybe you need to query one data source and then query another based on previous result. IN clause maybe a problem here since MySQL is not very good at it and Oracle even have a 1000 limit.
  4. API version controls that supports multiple versions.
    1. Maybe you simply need multiple sets of things completely (including different databases for different versions. Let’s face it, different versions probably require different db schema) and you have another service layer on top and when a request comes in, you call call one version easily but if it is a write request, you need to convert the request to make sense for other versions and make additional requests. Still this poses a big problem for data out of sync.
    2. Maybe you have multiple tables for different versions but the transaction will update all of them. Less problematic because it is in a transaction?
  5. Automatically generate documentations.
  6. An admin UI for modifying the data
  7. CLI for the following admin stuff:
    1. Configure database connection results in configuration file. Commit to git? Maybe only the connection and configuration but not the credentials.
    2. Create new edit existing table/resource results in schema migration and data migration files. Commit to git? Or should this be in database? Or should the schema migration be in git and data migration in database.
    3. Automatically have createdAt and updatedAt ?
    4. Optionally have complete history on some table? I don’t mean SQL history. That is in a big SQL log already. But history with who modified what. Very useful for audit. How does this work when schema changes?
    5. Use existing db through reflection. But this may not be necessary. Or this could be the killer feature.
  8. Maybe a JSON REST API and admin UI for these admin work above. But keep in mind that when there is an API and UI, how can we commit to git for the changes? Maybe should learn from the versioned wordpress.
  9. ACL Access Control Level. This may exists in a completely different layer on top of what we have above. We are not going to create a mysql user for each user of the system to leverage database’s access control, that will create a lot of connections to the database. So although ACL data is saved in database but it has to be the service that make sense of it and grant or deny access.
  10. Customize business logic.
  11. Support NOSQL and other types of datasource.
  12. Support multiple and multiple types of datasource together. I definitely feels better if this is another layer on top.

Docker: compare to virtual machine part 3

In my previous post, I mentioned that Microsoft is embracing docker, partnering with them, totally jumped in to this Docker wagon. As a former Microsoft employee, that was really kind surprise to me. I feel Microsoft has become really a different company with a new CEO. Also, to correct 2 of my friends working in Microsoft: NO, Docker does not support native Windows yet as of 5/30/2015. It will though, see here.

In this post, I’m going to talk about another company that really surprised me in this Docker world – CenturyLink. Yeah, you heard me, the CenturyLink cable company that did a lot of cheesy ads on $19.99 high speed internet for 5 years which is actually DSL and bundles with phone service. I used their DSL couple years ago. The experience was in one word: nah.

But, I have no clue what got into them or maybe their Innovation Lab is a completely separated entity, they seems to be a big player in this new Docker trend. I read quite a few good articles from them about Docker and watched several interviews the conduct with key person from Docker Inc. And more related to this post, they have this amazing tool that can really help you understand what a docker image looks like. Let’s load the sample images and see what we can learn.

Sample docker image layers

Sample docker image layers

Now, image these are cakes that are upside down. Each cake have different layers but they may have the same base. And the baker can bake one huge base and then put different stuff on top of them to become different cake. Some cakes can be very similar and share more than one base but several layers. In the picture above, ruby, python and node have the first 5 layers exactly the same. Only start to be different on the 6th layer.

Unlike a cake, and unlike virtual machine, the base layers is re-used in docker and it not reused by copy paste but reused as in there is only one copy of that layer needed. So say if you have python already on your machine with all the layers python need, and now you want to get ruby, it only need to pull down 5 layers instead of 10 since you already got the first 5 layers of ruby. Compare this to a virtual machine. Say you have a virtual machine running Windows, and then on top of it Microsoft Word, now you want a virtual machine running Windows and Microsoft Excel, you have to create a virtual machine image that have a completely separated Windows instance.