Recently, I have been working on a project for a client of mine connecting to REST APIs of PayPal. I must say that PayPal APIs are not the most developer-friendly and comfortable to work with. I encountered hiccups in webhook configuration, since there are multiple generations of them (“Webhooks”/IPN/PDT) and the documentation is lacking in clarity. Through trial-and-error somehow I found the right configuration. Then I came across a feature (a bug) in the transaction search API, that the payments only appear in the search results several hours after their execution.

There’s a reason, why many people looking for a payment provider, which is easy to integrate, choose Stripe. While Stripe offers an amazing official Go library, Go developer support on the PayPal side is weak. The only help comes from a community-made client for PayPal REST APIs.

My contribution to improve the PayPal client is a pull request with new transaction search method, which got merged in version v3.0.18. To anybody making a decision to about a payment provider, as long as it’s an option, I recommend to use Stripe.

Tags: Go, paypal, REST, stripe.

Who hasn’t had hers/his plants die at home while traveling for a few days? Unfortunately my plants suffered from this fate too. Since I already have my plants connected to the internet via Raspberry Pi, notifying me whenever the soil gets dry, one solution is to have that same event trigger actual plant watering. That’s what I set out to accomplish. By researching online, I stumbled upon an interesting tutorial Raspberry Pi Automated Plant Watering with Website which describes the hardware build very well, but adds a web server on top, which I find useless. I decided to run my own software stack.

Hardware parts necessary (assuming Raspberry Pi plant sensor is built already):

• mini motor water pump 3-6V
• 7-10mm aquarium hose
• container which can hold some water
• soldering iron, solder
• 5V relay
• old USB charger

After getting all the necessary parts, first step is to cut the USB charger phone part off and expose red and black wires. Next, solder the red wire sticking from the pump to the red wire exposed from the USB charger cable.

When the soldering is done, it’s time to connect everything together. See the complete wiring diagram with two moisture sensors in the figure below.

I have two sensors for two plants, but only one water pump for now.

Ports (as per pinout.xyz) used for the pump are (first relay, then Raspberry Pi 3):

• IN1 to BCM port 27 (pin 13)
• GND to ground (pin 6)
• VCC to 5V power (pin 4)

The next step is to fill the water container with water, tape the pump as close as possible to the container’s floor and use something (e.g. a wine bottle) to hold the outlet from the hose. I measured, that the motor can pump 23ml of water each second, thus it should be enough to pump for 1 or 2 seconds each time when the soil is dry.

As for software, get my updated scripts from the associated GitHub repo. The configuration from moisture.py can be seen below. First the moisture sensors are set-up:

# Define the GPIO pin that we have our digital output from our sensor
# connected to and URL which to call on sensor state change
sensors = {
        "peace_lily": (17, "** URL 1 **"),
        "parsley": (23, "** URL 2 **")
        }

then the water pumps:

# Define the GPIO pin that we have our digital input to the
# respective relay connected to and number of seconds to pump
# each time
pumps = {
        "parsley": (27, 1)
        }

When I get my second water pump, after wiring, I will just add a new entry under “pumps” with the corresponding pin.

Watering system in action

That’s all folks, happy farming!

Tags: iot, raspberry pi, smart plants, watering.

As it turns out, several people use my Feiertag Alexa skill, which I’ve described in a previous article. Recently I’ve upgraded it to use Amazon’s new device location API. Therefore now immediately after you install the skill through the Alexa app, you get asked, whether you want to share device country and postcode with the skill. That makes the user experience much easier, since you are no longer required to link an Amazon account to the skill. For backwards compatibility reasons, I’ve left the account linking in, but made it optional. So current users don’t have to change anything and will still get the correct Feiertag for their Bundesland.

Check out the updated skill in Alexa Skill Store. Just install it and ask: “Alexa, frage Feiertag, wann der nächste ist.”

Source code is in my GitHub repo.

Tags: Alexa, Amazon, AWS, iot, Lambda, Node.js.

I have a couple of pelargonium plants on my balcony and I’m always struggling to determine, when to water them. By chance I also have a Raspberry Pi 3 lying around, so I’ve decided to do a little IoT project, turn the plants into cyborgs and let them tell me, when they need watering.

The connected plant

The basic setup is inspired by the Moisture Sensor Tutorial. But instead of sending emails and needing an SMTP server, I just submit the data to ThingSpeak, a very useful IoT analytics platform. An event in ThingSpeak can be configured every time a differing value is received in the input data stream. I’ve configured such an event to trigger sending a tweet, check the plant’s Twitter account here. On my phone, I’ve then tweaked the notifications in the Twitter app using my personal account to notify me on every tweet on the plant’s account.

Raspberry Pi 3 with Witty Pi 2 mounted on top

To provide electric energy, I bought the Ravpower powerbank with capacity 26800mAh. However even the massive battery doesn’t last more than three days in this setup. That’s when I found a nice extension board for Raspberry Pi called Witty Pi 2, which will handle turning the Pi on and off based on a schedule, thus conserving huge amounts of energy. See the photo on the left. I set the schedule to turn the Pi on for 2 minutes once every 2 hours. The whole thing can keep running like this for about a month without recharging.

To summarize, here is a complete list of used components:

• Raspberry Pi 3
• Witty Pi 2
• Ravpower powerbank with capacity 26800mAh
• Moisture sensor
• Breadboard
• 5 jumper wires

Placement in a drawer on my balcony

Raspberry Pi 3 with Witty Pi 2 on top and a powerbank

ThingSpeak chart

The hardware running smoothly, finally, let’s discuss the code. I started off with the tutorial’s Python sample and extended on that. I ended up with Ansible scripts. Ansible takes care of the configuration deployment, so whenever I’d format the Pi, I just need to install Raspbian, replay the Ansible playbook and I will get the same software installed and configured on it. Check the repo on Github. Pretty sweet!

Tags: iot, raspberry pi.

The topic of AI assistants is recently one of the hottest in technology. With Apple’s Siri and “Ok, Google” now available in every smartphone, Amazon is not behind and has its own Alexa. As Amazon is not a phone maker, the assistant’s main place to live is a special speaker with integrated microphone array known as Echo.

In December 2016, Amazon was just introducing Alexa in Germany and came up with a promotion, under which everyone, who publishes three Alexa skills in the Alexa store within one month, will recieve a free Echo Dot device. That and curiosity were good motivation for me to start developing. I made a first primitive skill quite fast. It took just two hours. I wanted my second skill to be more advanced though. An idea came to me to make a skill, which tells you, the user, when the next public holiday is. That is actually quite useful in Germany, as different Bundeslände have different public holidays.

Alexa skill’s main part consists of an AWS Lambda function, which takes input from the Alexa speech model and outputs a text which will be told to the user. Lambda currently supports Java, Python and Node.js. As the developer gets charged by the used memory, I believe it’s best to use Node.js. Most of the examples online are implemented in Node.js too, and Alexa’s Node.js SDK  provides more features than the Java SDK.

My new skill, which I call Feiertag, requires you to link your Amazon profile and fetches the postal code from your primary Amazon postal address. The postal code is then used to determine the  Bundesland in which you’re residing and subsequently the public holidays that affect you. One of the ways to can interact with the skill is:

Alexa, frage Feiertag, wann der nächste ist.

As it took me a bit longer to finish the skill, I missed Amazon’s promotion for a free device. So when an Echo Dot became available for pre-order, I bought one myself. Unfortunately while testing my skill, I discovered, that the Alexa itself can tell you the holidays without the need to install any skill. You just ask:

Alexa, wann ist der nächste Feiertag?

This sentence sounds much more natural than the previous one, right? The difference is that it doesn’t contain invocation of my “Feiertag” skill. Too bad, that my skill is not bringing any new cool features, but at least I got to do a nice programming exercise.

See the source code of Feiertag skill on Github.

Tags: Alexa, Amazon, AWS, iot, Lambda, Node.js.

Last weekend we got together with Mark and Merten for a hackathon weekend with a goal to make a game. We have succeeded and the result is called Spaong. It’s about two alien robot races, Vulgarians and Profanians, who both hate humans. However a human astronaut seems to have gotten lost in the empty space between their planets! What will happen? The game requires two players and at least one of you needs to bring a gamepad.

Play the game at spaong.zee.cz to find out!

Tags: game, javascript, phaser, pong, space, spaong.

Expanding on the idea of my previous project MrTweety, I wanted to provide better real-time data and also try out big-data processing software. That’s when I discovered Apache Spark, toolbox used to manage distributed batch processing. My chosen project was not too complex, I was thinking of it being more like a proof of concept. The basic idea is to analyze a real-time feed of tweets posted globally on Twitter. From each individual tweet, occurrences of hashtags are counted and summed in 10 second intervals together. From these intervals a 15 minute sliding window is generated and hashtag counts again summed to form the final table. Top five most occurring hashtags are shown as a result.

To start, I checked out the Twitter API and saw that they offer streaming API. After some research, I’ve found an official Twitter Hosebird Client library and an example of how to use it to publish messages to Kafka. Kafka is  messaging system that is also a perfect source of data for Spark. That means we have one subproject for taking the Twitter stream and “producing” it into Kafka. A Spark subproject is then receiving the data and processing it. While working with Spark, I got a little derailed, when I found that you can use Twitter receiver directly with Twitter DStream. However when I actually tried this library, it didn’t work. I suspect that the DStream receiver tries to connect and read Twitter API from multiple threads and that forces Twitter to block the connections altogether. The Twitter DStream project got deleted from Github, while I was using it, so I’ve decided to go back to the Kafka approach. The official Hosebird Client claims to implement the limits and backing-off mechanisms correctly, so I believe it’s better anyway.

After that. Spark does all the hard computing work for us. We just specify, what we want to do with the data. Spark is made to be distributed, but for this small example we use just one node. Everything is running on Amazon ECS, which is a nice “Docker as a service” platform. To make it work, we just need an EC2 instance to run the Docker container. The basic t2.micro instance’s 1GB of memory was not enough for the all the included hungry JVMs, so I had to get a “spot instance” m3.medium with 3.75GB RAM. In the current market the costs should be around 7.5$ per month.

I’ve developed the whole project from scratch on Github as open-source, so check it out there. The production site is feud.eu. Here is a screenshot:

Note: A similar project that I’ve found and that could be interesting to check out is Twitter Analytics with Spark.

Tags: Java, spark, Twitter.

I came across this interesting piece of technology called Akka and I’ve decided to write a few words about it. So what is Akka? In short it is a toolkit and runtime for concurrent and distributed applications. To describe in more detail, we have to first talk about the motivation, why would someone use it.

Today we live in the era of distributed systems, billions of people use internet for their everyday life and successful online services have to plan for massive traffic. As soon as a service becomes popular, it is not economically viable to serve it using one large super-computer, but rather use a cluster of cheaper machines. As a programmer, who comes from a Java background, I have implemented concurrent thread-safe programs too many times to know, that it’s not easy. Concurrent systems introduce a new dimension of difficulty. To simplify the concurrent programing and introduce a common standard, company called Lightbend (previously Typesafe) came up with the Reactive manifesto.

Point of the Reactive manifesto is that distributed systems built following it’s guidelines will be easier to develop, scale, maintain and enhance at a later time. They are flexible, loosely-coupled, tolerant of failure and can handle unexpected behavior gracefully. Last but not least they they provide user with interactive responsive feedback.

Reactive traits. Arrows show, how they depend on each other.

So what are then the problems of concurrency? Distributed code by default behaves non-deterministically. Without any locking, every time you run the program, you get a different result. It is caused by multiple threads sharing a mutable state, e.g. a common global variable. That leads to race conditions, when things are happening in a different order than the programmer expected. Another problem is thread starvation, when a thread will never get to work and advance the states, because others have higher priority. This happens for example in an OS process scheduler, when there are many processes with high priority and a low priority process will never get to run on CPU.

Akka is a toolkit and runtime for simplifying the construction of concurrent and distributed applications on the JVM. Reasons to write Akka were practical. First simple actor implementation (not yet Akka) was written as a part of Scala in 2006. By 2008 Scala was used in complex server applications and need for a better library support was growing. That motivated developers to come up with Akka. It was released in 2010 and drew inspiration from Erlang’s libraries. Today it is a part of Scala standard library. Java programmers don’t have to be sad though, as there are also language bindings for Java. The library is open-source (Apache License 2.0) and developed by Lightbend, which is well known for its Play framework.

Akka has many features. These are the main ones:

• actor system, which introduces a concept of actor as a unit of execution,
• remoting allows messages to be sent between machines,
• routing defines how are the messages sent to the routees,
• clustering allows cluster nodes to run a part of a bigger application,
• Akka Persistence introduces actors with state, that remains even after restart, and
• Akka Streams provides asynchronous stream processing with non-blocking back pressure.

Actors encapsulate state and behavior. They are isolated and lightweight, meaning that they don’t have hard references to each other, only so called ActorRef. To communicate, they pass serializable messages back and forth. Advantage of this approach is location transparency – actors can communicate locally or over network, it’s only a matter of configuration. Akka is distributed by default, meaning you can only pass messages and everything is asynchronous. It hides all thread-related code behind its actor model and allows programmers to write code easily scalable and fault-tolerant.

We already mentioned you can use Akka in Java. To do that, you just have to make sure that all messages are immutable and serializable, because they could be sent over network. Immutability is achieved in Java by making all attributes final. Akka also requires toString(), equals() and hashCode() methods to be implemented.

When building reactive applications in Java, it is often convenient to use Lambdas. Lambdas are small blocks of code, that can be executed in separate threads. Typically these applications use many callbacks and Lambdas make it easy to write them. Before Java 8 we had to use inner classes and that was not as nice.

To see how can Akka be used, I recommend trying out these two examples:

https://www.lightbend.com/activator/template/hello-akka-java8
https://www.lightbend.com/activator/template/akka-sample-remote-java

Go to akka.io to read up on the documentation or to lightbend.com/activator/templates for more samples. Also check out Apache Spark, a fast and general engine for large-scale data processing built on Akka or Apache Spark Streaming, which can do the same for near real-time small batches of data.

Tags: akka, Java, reactive, scala, spark.

Hello dear visitor and welcome to the English version of my blog. The main purpose of this blog is to store memories of projects I did or experiences I had. You can find here mainly scripts, code samples and solutions to problems. Because of that, do not expect me to write posts every day, week or month. The point is to have a central and public location for my documentation.

The older articles are available only in Czech. Feel free to browse through them too!

.

V létě 2011 jsem na tomto blogu popisoval, jak vzdáleně aktualizovat WordPress. O tři roky později znovu řeším stejný problém, situace se však změnila. Dříve jsem měl na server přístup pouze přes FTP, nyní mám SSH.

Myšlenka nového skriptu spočívá v tom, že udržuje zdrojový kód WordPressu i jeho pluginů v SVN pracovních kopiích. Jak se to dělá pro samotný WordPress je popsáno zde. Poprvé je nutno provést SVN checkout nejnovějších tagů jádra WordPressu a jeho pluginů z repozitářů na http://core.svn.wordpress.org resp. http://plugins.svn.wordpress.org. Pro kompletní aktualizaci pak už vždy stačí spustit skript. Sám si zjistí tagy odpovídající nejnovějším verzím a pracovní kopie do nich přepne.

Skript si můžete prohlédnout:

#!/bin/bash
##
## Updates Wordpress in this directory to maximum release version.
##
 
LCYAN="\033[1;35m"
NOCOL="\033[0m"
 
update() {
  svninfo=`svn info 2>/dev/null`
  # check that directory is a working copy
  if [ "$?" -ne 0 ]; then
    echo -e "$LCYAN$(readlink -f .)$NOCOL is not a working copy."
    return
  fi
 
  oldurl=`echo "$svninfo" | grep '^URL:'`
  oldurl=${oldurl##* }
  oldversion=${oldurl##*/}
  tagurl=${oldurl%/*}
  newversion=`svn ls "$tagurl" | sed -e 's/\/$//' | sort --version-sort | tail -n 1`
 
  if [ "$oldversion" == "$newversion" ]; then
    echo "$(readlink -f .) is up to date with version $newversion."
  else
    echo -e "Update $LCYAN`readlink -f .`$NOCOL from version '$LCYAN$oldversion$NOCOL' to '$LCYAN$newversion$NOCOL'?"
    select prompt in "Yes" "No but continue" "Abort"; do
      case $prompt in
        Yes ) break;;
        "No but continue" ) return;;
        Abort ) exit;;
      esac
    done
 
    echo ---
    svn switch "$tagurl/$newversion"
    echo ---
  fi
}
 
if [ "$#" -ne 0 ]; then
  echo "Usage: $0"
  exit
fi
 
# main
update
 
# plugins
pushd wp-content/plugins/ >/dev/null
for plugindir in *; do
  if [ -d "$plugindir" ]; then
    pushd "$plugindir" >/dev/null
    update
    popd >/dev/null
  fi
done
popd >/dev/null

Také si skript můžete stáhnout.

Tags: bash hosting.