Why and How we are using Scala at journi

Last year we started building our web/mobile startup journi (called miavia back then), and I considered 3 different backend languages before finally settling on Scala:

  1. PHP. I didn’t want to continue working with a language that had grown into a monstrosity.
  2. C# .NET. Solid, stable language with functional programming features, but I wanted an open source solution instead, because I found Windows Servers less affordable and more difficult to automate than Linux Servers.
  3. Java. The JVM is almost as solid and fast as the .NET CLR, but Java was lagging 5-10 years behind C# in language features.

Enter Scala.

Scala_logoI had worked with Scala in 2010 and found it interesting, but at the time it seemed unusable for production. This had changed when I came across Play Framework 2 in 2013 and found that it was exactly what I needed.

I am not a language advocate, so don’t be surprised that I’m not trying to convince anyone to use Scala. But after 18 months of using Scala we found that:

  • Handling collections and options is very natural and readable, as control flow does not depend on exceptions and generally needs fewer tests.
  • Generally I found it easier to understand and write correct code that uses immutable variables and a functional programming style.
  • As the project grows bigger, we can more easily move forward with changes and refactor things. If the project compiles after a lot of renames, deletions, and so on, we have a lot of confidence that the code is still correct.
  • Scala’s Futures API makes it easy to interface with third party API endpoints while dealing with network timings and exceptions.
  • We use a lot of existing Java libraries. Very little work is needed to integrate with third-party services, because of the existing Java ecosystem.
  • Where performance is needed, you can easily get it. Sometimes, the most readable code does not perform as needed. There are a lot of monitoring tools for Scala out there. They can profile your app in production and give you detailed performance characteristics of your methods.

scala_pattern_match_memeThis outweighs the disadvantages:

  • In the beginning, newcomers from Java won’t immediately grasp how to use Options and other monads. Learning functional style of programming takes time, especially knowing when to use which paradigm.
  • You need a fast computer in order to have decent compile times. This is only a problem when checking out new code that has a lot of changes. While developing, the incremental compiler is quite fast.

Sometimes, because there are so many stable Java frameworks and new Scala frameworks, you spend more time evaluating which library to pick for basic things like database access. I’m pretty sure this will improve as the community matures.

I get asked quite often why we chose Scala in the backend, so this blog post serves as a reflection on this topic. Feel free to comment if you have more questions :-)

Posted in Play framework, Scala, Startups | 2 Comments

Scala Akka Workshop – Evolving a Robot in an Actor Cluster

It is quite extraordinary what is possible nowadays with a group of interested developers and powerful tools such as Scala and Akka.

Three months ago we had our first Akka Workshop here at the Vienna Scala User Group (VSUG), in which 25 developers participated in a real-time networked programming challenge. They had to compete in the non-zero-sum game of Prisoners Dilemma, which was powered by a central host, and network communication implemented with Akka.

Building an Akka Cluster

In the second workshop I wanted to raise the stakes a bit: what if we could make use of Akka’s clustering features? What if we could develop a cluster and a distributed algorithm on top of that cluster, during a 3 hour workshop? Continue reading

Posted in Akka, Play framework, Scala, Slick | 6 Comments

Play Framework Evolutions with Slick 2.0 Code Generator

The new Slick 2.0 database library comes with a simple code generator that generates Scala code for all your tables so you can quickly start coding against your database. It takes into consideration not only the tables and columns, but also primary keys and foreign key relationships.

Integration with Play Framework Evolutions – No More Boilerplate

So I thought it might be interesting to integrate this code generator with the Play Framework 2.2. Play has an Evolutions Plugin that takes care of database migrations. A basic evolutions file looks like this: Continue reading

Posted in Play framework, Slick, Web Development | Tagged , , , , , | 6 Comments

3 Things Learned from using Akka in a Coding Workshop

Here are 3 learnings from last weeks Akka coding session, in which all participants connected their Actors in order to participate in a Prisoner’s Dilemma competition. Continue reading

Posted in Akka | Tagged , , , | 4 Comments

Scala coding competition: Akka and the Prisoner’s Dilemma

More than 20 people with no prior Akka experience, 10 minutes preparation talk, one idea: getting everyone to write Actors participating in a realtime competition to beat others in the game of Prisoner’s Dilemma.

highres_309055212

Sounds challenging?
Yes, it is.

Did we manage to do it?
Yes, we did!

Before I explain what we did during the workshop, big thanks to the Scala Vienna User Group organizers and participants (special thanks to Rafael and Manuel for helping out), and to Stockwerk for hosting us! Also, Heiko Seeberger, Director of Education at Typesafe, came all the way from Munich – thanks for joining the workshop!

Actor what?

The workshop was split into 3 steps: First, I quickly introduced the concept of Actors and provided some code examples. And then the first challenge: develop your first actors in preparation for the game of Prisoner’s Dilemma. Once everyone was able to code basic Actor communication with Akka, we could move on to the final part: get everyone connected.

Real time tournament of Prisoner’s Dilemma

highres_309055922Using actor communication, all participants connected their Prisoner actors to the Tournament hosting actor, participating in a real-time round robin tournament. The tournament system continuously picked actors from the pool to play a game of Prisoner’s Dilemma against each other. Each actor represents a Prisoner in the game, and has to decide whether or not to cooperate with the other actor. Making the right decisions gives you points in the game, and we displayed the results live on a highscore.

Prisoner’s Dilemma explained

The game of Prisoner’s dilemma is actually pretty simple. Both prisoners have to make a decision. Each of them knows the name of the other prisoner, and has to decide: do you cooperate or defect? If both cooperate, they get some points. If both defect, they don’t gain anything. But if one cooperates, and the other one defects, the cooperator looses points and the defector gets it all, exploiting the other prisoner.

An Evil Twin strategy

If you are a prisoner in the Prisoner’s Dilemma, you better remember who the bad guys are, otherwise you’re not going to make a lot of points in the game.

After running the tournament for a while, one participant using an “Evil Twin” strategy scored most of the points. Peters strategy was straightforward: Be nice for a while, but then switch strategy and try to exploit everyone for a while, before turning back to being nice again. It’s quite challenging to implement a strategy like this in just a few minutes time. So, congratulations Peter! You can see his Evil Twin code here on GitHub.

Connected Actors for the win – what’s next?

highres_309055912The gamification aspect of the workshop was a lot of fun. Everyone could develop different strategies and watch how they play out against the other ones in real time.

Now one question remains: what’s next? If you have some ideas, some games perhaps, or other things that could be implemented with live-feedback during a workshop, please let me know! Comments and suggestions are always welcome.

Update [27.11.2013]: Check out our learnings from using Akka in a coding workshop

Links:
[1] Code of the Play framework application that hosted the tournament during the workshop (Akka+Play+Websockets)
[2] Meetup page of this workshop
[3] “Evil Twin” solution by Peter

Posted in Akka, Play framework | Tagged , , , , , , | 2 Comments

Using Play Framework 2 in production

Even though Play Framework 2 is a young framework, we found it to be quite stable for production use. Still, here are some points you may want to consider when working on a serious web project using Play 2.2.

This is a series of blog posts touching different aspects of web development and deployment. Here’s the list of topics:

1.) HTTPS support using Nginx

If you provide some sort of user login on your page, you’ll probably want to use HTTPS/SSL. This is a how-to for using Play with an nginx reverse proxy.

2.) Database access: Integrating Slick 2.0 with Play Evolutions

If you’re using a relational database such as MySQL, you’re probably also using Play’s Evolutions Plugin for database migrations. This article shows how to integrate the new Slick 2.0 code generator with the Play Framework so that code for querying your tables is automatically generated.

3.) Logging (coming soon)

By default, there will be only one log file created by Play, this may not be what you want on your production server. How-to coming soon.

4.) Developing with Play: CSS/less compilation (coming soon)

If you have lots of stylesheets, or when using Twitter Bootstrap, the built in less compiler can become annoyingly slow during development. Thankfully there is an easy workaround using the faster lessc compiler. How-to coming soon.

5.) Developing with Play: Javascript reverse routes (coming soon)

Play can automatically generate javascript reverse routes which can be used in your javascript front-end. In production, you will probably want to minify and compile your javascript to one big file for performance reasons. How-to coming soon.

 

Posted in Play framework, Slick | Tagged , , , , , | 3 Comments

Play Framework 2 in production: HTTPS using nginx

If you provide some sort of user login on your page, you’ll probably want to use HTTPS/SSL. Play Framework 2 supports secure connections using the recommended setup with a reverse proxy such as nginx.

This means your application server will run your play framework application on a private port (the default is 9000), as well as an nginx web server which listens to the public ports (HTTP/80 and HTTPS/443), and redirects those requests back to your play app.

Nginx Configuration for HTTPS

Installing nginx on most linux distributions is usually a one-liner (please refer to the nginx documentation). Here is the configuration we use for setting up nginx as the reverse proxy. In this example we assume that your domain name is example.com and that your compiled Play application files (using play dist) are located in /var/example/target.

Nginx config file: /etc/nginx/sites-available/default

# Redirect everything to HTTPS and all subdomains to example.com
server {
	listen 80;
	example.com *.example.com;
	rewrite ^ https://example.com$request_uri? permanent;
}

# HTTPS server
server {
	listen 443;
	server_name example.com;

	# play public assets
	root /var/example/target/scala-2.10/classes/public;

	# SSL configuration
	ssl on;
	ssl_certificate /var/example/target/deployment/example_com.pem;
	ssl_certificate_key /var/example/target/deployment/example_com.key;

	ssl_session_timeout 5m;

	ssl_protocols SSLv3 TLSv1;
	ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv3:+EXP;
	ssl_prefer_server_ciphers on;

	# static content (with client-side caching enabled)
	location /assets/ {
		access_log  off;
		log_not_found off;
		add_header  Pragma "public";
		add_header  Cache-Control "public";
		expires     30d;
		alias /var/example/target/scala-2.10/classes/public/;
	}

	# serve assets or request page from proxy (if asset not found)
	location / {
  		try_files $uri @proxy;
	}

	# the play web server
	location @proxy {
		proxy_pass  http://localhost:9000;
		proxy_redirect off;
		proxy_buffering off;

		#send protocol info to play server
		proxy_set_header        Host               $host;
		proxy_set_header        X-Real-IP          $remote_addr;
		proxy_set_header        X-Forwarded-Proto  https;
		proxy_set_header        X-Forwarded-Ssl    on;
		proxy_set_header        X-Forwarded-For    $proxy_add_x_forwarded_for;
	}
}

Checking for SSL in your play app

Note that in this example configuration, ALL traffic will use HTTPS. If you want to use both HTTP and HTTPS, you’ll have to remove the redirects from http to https. You can check in your play app which protocol the request uses:

def someAction = Action {
	implicit request => {
		val isSecure = request.headers.get("X-Forwarded-Proto").exists(_ == "https")
	}
}
Posted in Play framework | Tagged , , , , , | 4 Comments

Recent challenges and outlook

Hello everybody,

it’s been quite a long time since my last post here, and that was partly because of my focus on running the technical implementation of our web application, an editor which lets you easily create and share insider tips for travelers.

However, the real challenge here at miavia is not web development, although implementing beautiful designs can be quite difficult. It’s not about how, but about what to do.

Currently, we want to get as many users on our platform as possible, and we want mostly content creators. People who create content are often quite careful about which tools or platforms they use, and rightly so. We try to listen as carefully as possible to the feedback we get, but sometimes it can lead into a direction we don’t want to go.

Part of the challenge comes from explaining what we want to create, and it has been a constant concern of ours to improve communication on the website and elsewhere.

We’re not interested in becoming a TripAdvisor, or a WikiTravel. The interwebs are full of information about specific sights and well known places.

We’re all about the great places you haven’t heard of yet. And while existing sites have some kind of social network integration, we believe that there are lot more possibilities to explore about the social aspect of traveling.

Now, a lot has happened since May and also our team has grown, as we are currently 6-7 people, three of us are software developers. This allows us to work on both, our web app and our upcoming iPhone mobile app, in which we want to take advantage of the upcoming iOS 7.

So: if you just came back from a great trip, head over to http://miavia.in and write about it! :)

image

Posted in Startups | Tagged , , | Leave a comment

A word on search engine optimization

There’s a lot of misinformation about SEO out there. Some stuff is plain wrong, other infos may have been true 10 years ago. We’re now planning on getting a lot of content online, so I wanted to get up-to-date information from people who have solved SEO problems.

So I just had a very interesting talk with Franz Enzenhofer from Full Stack Optimization.

Google tries hard to filter out duplicate content, so having unique content on our site gives us a clear advantage and hopefully a better rank than services that just crawl other pages and copy their content.

Also very interesting is the fact that Google tries to find out how a page looks to the user. So it renders CSS and, contrary to popular belief, does execute some Javascript when crawling. It even executes some asynchronous AJAX requests, although it might do so a day later. Does this mean we can throw away server-side HTML generation and just use client-side rendering frameworks like AngularJS and Ext JS to make single-page applications that are search engine optimized?

Unfortunately no. Unfortunately not yet.

While Google does execute some JavaScript snippets, it will probably choke at any application that’s too complex or too large to download. Also, it must be the case that each page is accessible via a unique URL.

Maybe in 5 years things will be different ;)

Posted in Startups, Web Development | Tagged , , | Leave a comment

Real-world performance of the Play framework on EC2

Recently there have been discussions about the performance of the play framework, after a large collection of web framework test results has been posted on TechEmpower.

Play does well on these tests, but it’s not exceptionally fast. In the article, simple JSON responses are tested on an EC2 large instance, one of Amazon’s faster virtual servers. According to the authors,

  • Play serves ~9000 requests/sec
  • PHP serves ~4000 requests/sec
  • Netty serves ~40000 (!) requests/sec

… just to name a few of the tested frameworks.

Netty was the fastest “framework” in the test, but it’s actually more a web server than a web framework. As a side note, Play uses Netty for serving HTTP requests.

But how how does that compare to a real-world scenario?

Testing what users (and their browsers) really do

Since we’re also using the play framework (2.1.1-scala) in production, we wanted to see how many pages per second we can really serve in our Amazon EC2 environment.

In contrast to the above article, the following results are obtained by load-testing complete pages that we use in production. This means that we download not only the HTML from the server, but also all images, CSS, Javascripts, like normal browsers do.

There were 3 different tests, run on EC2 micro and large instances. (Look how much you can do with a micro instance!)


Test 1 – Small page with database access

A small page with CSS, Javascript, and a picture. It also selects data from a MySQL database, but it’s a simple indexed query. We use Slick for database access.

  • Total page size: 30 kb
  • Total # of requests per page: 4 (Html, Css, Javascript, Png-Image)
  • EC2 Micro Instance result: 30 pages/sec  (120 requests/sec)
  • EC2 Large Instance result: 600 pages/sec (2400 requests/sec)
  • Bottleneck: CPU


Test 2 – Page with lots of content

A larger page with CSS, Javascript and lots of pictures. No database query here. The page is multi-language and uses Play’s Internationalization feature.

  • Total page size: 900 kb
  • Total # of requests per page: 15 (Html, Css, Javascript, 10 Png-Images)
  • EC2 Micro Instance result: 8 pages/sec  (120 requests/sec)
  • EC2 Large Instance result: 90 pages/sec (1350 requests/sec)
  • Bottleneck: Disk/Bandwidth


Test 3 – On-the-fly server-side image resizing

In this test, just one request is made. The server loads a JPEG image from an Amazon S3 bucket, resizes it from 800×600 to a 300×225 thumbnail, and sends it again JPEG-encoded to the client.

  • Total response size: 12 kb
  • Total # of requests: 1 (JPEG-Image)
  • EC2 Micro Instance result: 4 requests /sec
  • EC2 Large Instance result: 160 requests /sec
  • Bottleneck: CPU (the image rescaling algorithm)


Conclusion
: We can serve quite a lot of users with the play framework, using its default settings. Because we did not optimize our code for the benchmarks, we could not serve as many requests as in the article on TechEmpower; but the results are in the same ballpark.

Surprising EC2 micro instance performance

The EC2 micro instances can actually sustain quite a high number of requests per second. If you need to serve 30 full page requests per second (like in Test 1) during peak hours, you’ll probably serve an average of 8 requests per second during the day (4 times less). That’s about 20 GB of data per day, which means you will actually pay an order of magnitude more for the traffic than for the virtual server itself.

(However, running a production server on a micro instance may not be your thing. AWS allows only around 16% of sustained CPU usage on these instances, meaning it’s trivial to DoS-attack your server.)

From an economic perspective, play’s performance means you will be able to focus on other issues, such as saving bandwidth ;-)

Posted in Play framework | Tagged , | 8 Comments