Bruno

Michel

Ingénieur étude et développement

Comme annoncé hier, j'ai fait une présentation intitulée « 45 projets en 45 minutes ». L'idée de cette présentation est de parler d'un large éventail de projets web ou dans des langages de programmation Open Source pour inspirer les auditeurs, leur donner de nouvelles idée, l'envie d'utiliser ces projets, voir d'y contribuer.

Vous pouvez regarder les slides ou juste consulter la liste des projets présentés :

• Twitter Bootstrap
• Foundation
• Jekyll Bootstrap
• c3po
• Masonery
• ShareJS
• ShareLaTeX
• Tinycon
• CSS Refresh
• Plax
• Luvit
• Rust
• Webrocket
• CodeMirror
• ditaa
• Dochub
• beautiful-docs
• documentup
• MailTrap
• Trello
• Gitlab
• ownCloud
• Sensu
• spdylay
• Elastic Search
• IndexTank
• Strano
• dreadnot
• Gatling
• Amon
• fnordmetric
• Redmon
• Cloud Foundry
• js2coffee
• Sugar
• Ember.js
• EventSource
• d3.js
• Socialite
• Leaflet
• Alice
• KendoUI
• Geeklist
• Slide-em-up
• LinuxFr.org

Comme vous le savez sûrement déjà, af83 a une participation très active dans les communautés, et notamment dans les événements qu'elle organise, sponsorise ou prend part d'une façon ou d'une autre. Voici quelques événements où vous pourrez nous rencontrer si vous le souhaitez.

Ce soir, quelqu'uns d'entre nous iront au meetup Paris.rb.

Demain, je serai présent au Microsoft TechDays pour présenter 45 projets en 45 minutes.

Ensuite, nous participerons au prochain Paris.js et au Paris Opa Devcamp #1 organisé par MLState, Placeloop et Geeklist (au fait, vous pouvez nous rejoindre sur geeklist).

Au mois d'avril, rien de sûr, mais nous avons proposé des talks à Devoxx et www2012.

Enfin, nous ne manquerons pas les Rencontres Mondiales du Logiciel Libre à Genève. Et il se murmure que nous pourrions organiser un prochain Web Workers Camp ;-)

En Ruby, on utilise très souvent les deux structures composites que sont les tableaux et les hashs. Mais la bibliothèque standard ne se limite pas à ces deux là.

Je fais toujours une grimace quand je vois du code qui ressemble à ça :

foos = []
data.each do |data|
  foo = transform(data)
  foos << foo unless foos.include?(foo)
end
do_something_with(foos)

Ce code construit un tableau d'objets sans doublon. Pour cela, avant d'insérer un élément dans le tableau, il vérifie que celui-ci n'est pas déjà présent dans le code.

Or, pour ce besoin précis, Ruby propose des Sets, c'est-à-dire un ensemble non ordonné d'éléments uniques. L'exemple précédent devient :

require "set"
foos = Set.new
data.each do |data|
  foos << transform(data)
end
do_something_with(foos)

Les Sets sont des Enumerables qui s'utilisent un peu comme des tableaux, mais qui évitent tout doublon :

require "set"
foos = [3, 2, 1].to_set
foos << 2
foos  # ~> #<Set: {3, 2, 1}>

En dehors de l'aspect pratique, les Sets sont également plus performants que les tableaux pour ce cas d'usage. Dans le cas du tableau, avant d'ajouter un élément, il faut parcourir tout le tableau pour vérifier que l'élément n'est pas déjà présent. Par contre, les Sets utilisent des Hashs pour le stockage et le temps d'insertion d'un élément ne dépendra pas du nombre d'éléments dans le Set.

Une variante intéressante des Sets sont les SortedSets. Ils sont similaires aux Sets mais garantissent que l'on accède à leurs éléments dans l'ordre. Ce sera sûrement plus clair sur un exemple :

require "set"
foos = SortedSet.new
foos << 3
foos << 1
foos << 2
foos << 2
foos.to_a  # ~> [1, 2, 3]

Voilà, j'espère que la prochaine fois que vous aurez besoin d'une collection d'éléments uniques, vous penserez à utiliser les Sets pour m'éviter une grimace le jour où j'irais lire votre code sur github ;-)

Recently I was asked to contribute to the devblog… Since I don’t code, Bruno suggested I’d talk about how to write nice content. So here are a few pieces of advice for your blog articles:

Find something to talk about where you actually have something to say – yes, you do have one: feedback on tools, languages, events you attended, opinion on methodologies, cool stuff you just coded, questions for the community…

Start with an MVP, that is to say your idea plus the main points, and only then begin writing up. Don’t be afraid of refactoring, you probably won’t say it the way you want to on the first draft.

KISS and DRY doesn’t only apply only to code. You’re not writing a novel, and there’s no reward for the longest post on the internet. So go straight to the point, and don’t lose your reader in phrases that look like paragraphs.

If it keeps being too long, maybe you’re trying to fit too much information in your message: use packets, and break it in smaller parts with indications for the reader to retrieve the other parts of the message.

Respect the language conventions so your post will be parsed correctly: grammar, syntax, spelling… You do it for programs why not for your visitor?

Run tests with willing proofreaders, there’s always bugs you won’t have spotted.

Keep practicing; you didn’t become a code ninja in a single night…

These best practices may seem obvious… But unfortunately they are not always implemented (yes, we all have some bad reading experiences in mind…), so let’s try and use them.

HTTP is the king of communication protocols, and it just won the internet war. Everything else is details; I love details.

Distant APIs

It has become almost indisputably common sense that non local services have to expose their APIs as REST APIs. From small private services to large cloud services, everybody speaks HTTP, the new lingua franca. You can debug with curl, handling authentification, compression, content negociation, caching and all other http wonders. But it has a cost, an overhead, and I don't even speak about SOAP here.

However there is a challenger for doing RPC without language restrictions.

Thrift was created by FaceBook for exposing services to different languages. It uses a neutral grammar to generate code for your language. The protocol specifies different communication layers (like TCP), rpc and errors; just like SOAP without XML and HTTP. It's a nice thing for class oriented developers.

You can do similar things with Google's Protobuf, albeit with more work, because only serialization and grammar are specified. There's also Avro, which is an attempt to simplify Thrift while cutting the link with Facebook. And it's more geared towards dynamic languages, Ilya Grigoric explains it better than me.

Minimalism

I wanted to try something distinct from HTTP. Something connected, with a simple message protocol, similar to communications between actors in the actor pattern.

Such a protocol is the memcache protocol. It's used to silently replace a memcache server (see CouchBase), or to expose a simple persistant connection (ElasticSearch).

When implementing a protocol used by many servers, you can use handcrafted and optimized client implementations...

So, if it works with Memcached, why not try Redis? The number of clients is huge, most of them based on hiredis, the official C client. And the protocol is extensible.

You can put a real Redis in your stack and mix Redis' patterns (queue, pubsub, incremental counter, cache…). All that while using a single protocol between application services and Redis services.

Expose your services as a Redis server

I chose nodejs to make some tests. Node is asynchronous and the king of POC; I could have chosen Erlang or EventMachine, but I wanted to see something running quickly.

The server is simple: a TCP server feeding a Redis parser. The parser is a hidden API, but node is not so strict. When a command is complete, a reply event is thrown, and you can handle the reply with the simple types available in Redis.

var redisd = require('redisd');

var server = redisd.createServer(function(command) {
    if (command[0] === 'info') {
        this.encode('redis_version:2.4.5');// I'm a liar
    } else {
        this.singleLine('OK');
    }
});

server.listen(4242, function() {
    console.log('server listening on port 4242');
});

You can test this server with redis-cli and your favorite language API. This project is small, it's just glue between well tested products. But now you can go ahead and build your own, with gevent, eventmachine, erlang … or implementing the MULTI/EXEC patterns...

If you are interested in how we made this year's new year greeting (which you can find here 2012.af83.com), here is a simplified architectural view:

Basically a simple Sinatra based server, serving the static content. When you submit the form it repeates the message on an irc channel using Cinch. There are two other IRC bots on the same channel, one is on the EEEpc controlling the Fux that Controls the Tux, which makes it speak and move. The other is on the laptop that is connected to the camera, adding the text from the IRC channel to the video which is then encoded and streamed to the Wowza which reencodes it for Flash and HTML5 video.

Sounds complicated?

Well it ain't, basically all of the server side code can be resumed to :

require 'cinch'
require 'sinatra'
require 'eventmachine'

set :public_directory, Proc.new { File.join(root, "public") }

get '/' do
 File.read(File.join('public', 'index.html'))
end

post '/form' do
  $bot.send("#af83-2012", "tux speak " + params[:message])
  $bot.send("#af83-2012", "tux mouth #{params[:message].split.size} close")
  $bot.send("#af83-2012", "tux flippers 2 down")
end

$bot = Cinch::Bot.new do
  configure do |c|
    c.server = "irc.freenode.org"
    c.channels = ["#af83-2012"]
    c.nick     = "TuxCamBot"
  end
end

EM::defer {
  $bot.start
}

Après le célèbre Twitter Bootstrap que l'on ne présente plus (voir l’avis de François), 99Lime HTML Kickstart fait son apparition, il s'agit d'un framework html / css / jquery qui permet de produire plus facilement et rapidement le layout de votre site.

Framework assez complet, documentation accessible, il commence déjà à plaire à beaucoup de développeurs et intégrateurs si l'on en croit les tweets récents. Au premier abord je le trouve d’ailleurs plus facile à prendre en main que Twitter Bootstrap.

Simple à installer et à utiliser, il vous réconciliera avec le html / css, on y retrouve tout ce qu'il faut pour démarrer rapidement un projet sans que les maquettes soient finalisées :

• Listes
• Menu/Navigation vertical gauche ou droite / horizontal avec des effets js pour les sous-menus
• Tableaux
• Des choix de boutons, avec ou sans couleur, avec ou sans pictos (même si je reste réservée pour les images dans les boutons)
• Onglets positionnés à gauche, au centre ou à droite
• Fil d'ariane
• Plusieurs dispositions en colonnes (taille et nombre)
• Différents designs, formats et placement de l'image, sans oublier une galerie d'images
• Slideshow
• Formulaires

Pour finir, oubliez les différences d'interprétation des navigateurs car le framework est compatible avec tous… enfin mention "dégradé" sur IE6 !

À votre tour : 99LimeHTMLKickStart

Ori Pekelman rajoutera sa Pekelman's Touch :

Autre petit intérêt, la grille proposée est fluide, à la différence de bootstrap orienté largeur fixe.

Par contre quelques bémols assez importants... lime n'as pas du tout été pensé comme bootstrap pour être une mince couche modifiable par la suite. Nulle variable. Pas de Mixins. Ni Less, ni SCSS, ni SASS.. Donc l'utiliser c'est de le forker...

Démarrage rapide mais maintenabilité très basse. D'ailleurs le fait qu'il propose un lien de téléchargement d'archive plutôt qu'un lien vers le dépôt github démontre bien qu'il s'agit d'un projet qui n'est pas vraiment orienté vers les développeurs structurés.

Le JS embarqué est de la même nature, plusieurs librairies ensemble dans le même fichier, et qui tapent en dur sur des sélecteurs comme ul.menu (d'ailleurs voir de nos jours <div id="nav"> et <div id="footer"> ça fait un peu mal, pas très HTML5 tout ça).

Sinon, si pour vous l'intérêt principal c'est la grille fluide, allez voir le très sympatique http://semantic.gs (qui d'ailleurs par la magie de Mixins, peut très bien fonctionner avec bootstrap).

Ajout : il existe maintenant htmlkickstart-rails pour utiliser HTML Kickstart avec l'asset pipeline de Rails.

Full-text databases can do more than full text search. Elastic Search provides a full toolbox for indexing and manipulating text and data. Tire provides a nice and rubyish interface for using this tool.

Autocomplete is one of its basic features. Autocomplete suggests words starting with letters that you have just typed, and refining suggestions as you type.

Iterate and filter

Most databases handle that feature with a filter (with LIKE keyword in SQL, regular expression search with mongoDB). The strategy is simple: iterate on all results and keep only words which match the filter. This is brutal and hurts the hard drive. Elastic Search can do it too, with the prefix query.

With a small index, it plays well. For large indexes it will be more slow and painful.

Look for what you want

Elastic search is an index, iterating over an index is a bit paradoxical. The main feature of an index is finding quickly something and giving back a reference to the complete information. Just like an index in a book: your eyes scan quickly a list of alphabetically sorted words, and you have the page number for reading the information.

It's hard to build an index, but when it's done, it's very fast to find something. Usually, data are not modified very often, there is far more reading than writing, the cost of indexing is cheap when you compare it with the searching speed benefit.

Elastic Search indexes document with tokens extracted from properties. Prefix search is a specific need and is handled with edge ngram, a variant of ngram. Ngram is a group a contiguous letters extracted from a word. Edge ngram is a ngram built from the start or the end of a word.

For example, you are a biologist and want to index the word Heterastridium, with a min size of 3 and max size of 6. Too few letter is not enough deterministic, too many is a waste.

This word Heterastridium is tokenized as:

• het
• hete
• heter
• hetera

Like any elastic search query, I can add more criteria, filters or facets.

Tire in action

Edge ngram tokenization is very specific, it can't be used for full search, or even sorting, but one property can be indexed more than one time, and elastic search handles that nicely.

Code example

Boilerplate.

#encoding: utf-8
require 'rubygems'
require 'tire'
require 'json'
require 'active_support/core_ext/object/to_query'
require 'active_support/core_ext/object/to_param'

One analyzer for starting, one for sorting.

conf = {
  settings: {
    analysis: {
      analyzer: {
        my_start: {
          tokenizer: 'whitespace', #I know it's a title, with no ponctuation
          filter: %w{asciifolding lowercase my_edge} #no accent, downcase
        },
        my_sort: {
          tokenizer: 'keyword', #cut nothing, it's just for sorting
          filter: %w{asciifolding lowercase}
        }
      },
      filter: {
        my_edge: {
          type: 'edgeNGram', #1 to 10 letters, from the start
          min_gram: 1,
          max_gram: 10,
          side: 'front'
        }
      }
    }
  },
  mappings: {
    coral: {
      properties: {
        id: {type: 'string', index: 'not_analyzed', include_in_all: false},
        name: { type: 'multi_field', fields: {# this property needs multiple index
          start: {
            type: 'string', analyzer: 'my_start', include_in_all: false
          },
          sort: {
            type: 'string', analyzer: 'my_sort', include_in_all: false
          }
        }
      }
    }
  }
  }
}

Some data : scientific name of corals stolen from Wikipedia.

corals = [
  'Hydractinia echinata',
  'Heterastridium',
  'Hydractinia symbiolongicarpus',
  'Hydrichthys'
]

Feeding an empty index with corals.

Tire.index 'corals' do
  delete
  create conf

  cpt = 0
  import corals.map{ |coral|
    cpt += _posts/2012-01-12-autocomplete-with-tire.md1
    {id: cpt.to_s, name: coral, type: 'coral'}
  }
  refresh
end

Most of tire's job is to provide a complex JSON to elastic search.

Searching and sorting. Which words start with "hydr".

s = Tire.search 'corals' do
  query do
      string 'name.start:hydr'
  end
  sort do
    by :'name.sort', 'asc'
  end
end

s.results.each do |document|
  p document.name
end

The code example prints the list of words in the standard output, quickly and sorted.

Conclusion

Brute force is alway an option, ad hoc tools can do more, with less.

Elastic search provides specialized tools for common needs, use them!

Look Mom, no object

I don't remember who sold me object orientation, but he was a great liar. — José Valim

In Erlang, there are no objects. It's a purist and functional language. Everything is made of primitives types; three differents kind of lists (list, tuple, binary), no decent string representation. It's one of the weird things when you discover Erlang. Weirdness and integrism are the beauty of Erlang: Erlang doesn't want to be kind with developers, and erlanguists like it. "No object" is not a motto, it's a strategy to handle communication efficiently. Everything is message, everything can go through a wire. There are no objects, but conventions: specific representation with specific functions. dict, array, digraph, gb_sets, gb_trees, lists, orddict, ordset, proplists, sets, string, unicode. All of them are in the standard library. Useful to manipulate, efficient in production, ugly to display. No syntax sugar or cute display, everything is a naked primitive object.

IOList

Now, you have unplugged "object pattern" from your brain. Plug in "list manipulation" and "pattern matching": Erlang is a functional language.

And IOList is one of Erlang's secret weapon. Although almost undocumented, you can find references to it in the documentation or while debugging. So, what is it used for?

List appending is a curse. You want to concatenate two lists? Build a new one with enough room. Copy each items from the first list, then each items from the second one. But when you concatenate inside a loop, the amount of copies becomes huge, and first items are copied again and again.

Not so efficient...

Erlang has a tactic to handle concatenation. When you want to concatenate two things, just put them in a new list - a new one for each concatenation. The result is no more a flat list, Erlang names it a deep list. And when your bag is full, you can flatten it with lists:flatten. You append many times, and flatten one time.

1> A = "Hello".
"Hello"
2> B = " world".
" world"
3> C = [A, B].
["Hello"," world"]
4> lists:flatten(C).
"Hello world"

However, APIs done right (io:format, file:write, gen_tcp:write…) accept iolist, so flattening is not mandatory.

Some functions return iolists, dont' be surprised:

1> Dump = io_lib:format("dump ~p~n", [{foo, "Hello words"}]).
[100,117,109,112,32,[123,["foo",44,"\"Hello words\""],125], "\n"]
2> lists:flatten(Dump).
"dump {foo,\"Hello words\"}\n"
3> io:format(Dump).
dump {foo,"Hello words"}
ok

In this example, an Erlang structure is dumped. When displaying hierarchical data, a deeplist (i.e. IOList) is returned. The five first numbers are ASCII value of "dump ". That's clear, it's raw data for robots, not for humans. It's more readable through lists:flatten, but use it raw in your code. When flattened, the shell interprets it as a string. Then, io:format accepts the rough iolist.

Erlang loves bags of numbers: everything is just compound of 8 bits brick, isnt'it?

Tire est une bibliothèque ruby pour effectuer de la recherche full text. Il s'intègre facilement avec Active model et utilise Elastic Search, une application REST basée sur Lucene.

Conçu pour tourner au sein d'une application Rails et d'Active Support, Tire peut aussi être utilisé directement, sans Rails.

Paramétrages

Tire propose un service qui fonctionne bien avec les réglages par défaut, mais qui exprime vraiment son potentiel avec des réglages plus fins.

Analyseurs, la théorie

Un document est découpé en éléments (tokens), qui sont ensuite filtrés. Ces éléments servent ensuite de clef pour désigner le document. On parle d'index inversé.

Le découpage est effectué par un tokenizer, le filtrage par des filters. Un tokenizer assemblé avec des filters est un analyzer.

Elastic Search rassemble des éléments issus des projets Lucene et Solr. Ces éléments ont un long historique, certains sont redondants, d'autres très similaires. De plus, beaucoup de ces outils sont prévus pour l'anglais... Les efforts pour gérer les autres langues sont très variables, et les différents styles de grammaires facilitent plus ou moins la création d'outils spécifiques.

Les filtres classiques ne sont presque pas destructifs, et se contentent de tout mettre en minuscule, d'enlever la ponctuation et même les accents. Mais les dégâts faits par un filtre seront plus ou moins importants selon les langues : enlever les accents en anglais est anecdotique, passable en français et un massacre en vietnamien. De toutes façons, l'indexation est plus une histoire de recette de cuisine mitonnée pour chaque projet qu'une vérité absolue et universelle.

Les "stop words", mots vides, sont des mots trop courants qui, isolés, ne désignent rien de précis - comme les articles, les conjugaisons d'être et avoir, ce genre de mots. R ou go sont deux exemples de mots martyrs de Google, très difficiles à rechercher. Il est possible d'étoffer sa collection de mots vides en piochant dans les mots ayant une trop grande fréquence dans son index.

Le "stemming" ou lemmatisation est un principe plus controversé, qui se base sur un ensemble de règles pour retrouver la racine d'un mot. Il permet de normaliser les singuliers/pluriels, masculin/féminin et autres déclinaisons d'un même mot. Dans certaines langues à déclinaison, ça a une bonne réputation, en français, c'est un peu moins convaincant. Snowball, un lematiseur bien connu s'en tire pas mal en regroupant cheval et chevaux mais sans les confondre avec chevalet et chevalier. Sur des noms propres en revanche, le résultat peut être plus fantaisiste... Et pour des recherches sur des termes précis, comme le titre d'une oeuvre, la lematisation amène des surprises.

De manière générale, en français, il faut se méfier des homonymes : "avions", c'est le verbe avoir conjugué, et donc un mot vide, ou le pluriel d'avion ? Avec les noms de marques et les termes issus de l'anglais, les collisions sont encore plus fréquentes.

Pour gérer des noms propres, il existe des outils qui essayent de convertir les mots en phonèmes. Mais les outils proposés par Elastic Search sont spécifiques à l'anglais et certains algorithmes ont presque un siècle, ce qui les rend peu efficaces en français. Ayons d'ailleurs au passage une pensée pour les administratifs qui appliquaient ces règles à la main pour le recensement...

De façon générale, pour couvrir différents besoins (recherche, tri), il n'est pas rare d'indexer plusieurs fois un même champ avec des règles différentes.

Enfin, quel que soit la solution choisie, pour éviter d'avoir des résultats fantaisistes, il faut utiliser le même analyseur pour indexer et pour rechercher.

Exemples concrets

#encoding: utf-8
require 'rubygems'
require 'tire'

Tire pense que le monde est Rails, deux imports permettent de contourner ça.

require 'active_support/core_ext/object/to_query'
require 'active_support/core_ext/object/to_param'

Paramétrages de Tire

tire = Tire::Index.new 'analyzers'
tire.delete
conf = {
  settings: {
    analysis: {
      analyzer: {
        francais: {
          type: 'custom',
          tokenizer: 'standard',
          filter: %w{lowercase stop_francais asciifolding elision}
        },
        francais_boule: {
          type: 'snowball',
          language: 'French'
        }
      },
      filter: {
        stop_francais: {
          type: 'stop',
          stopwords: %w{je tu il nous vous ils le la les un une des a ai et est ayons ça}
        }
      }
    }
  }
}
tire.create conf

Test de différents analyseurs avec du contenu.

%w{ whitespace standard simple french francais francais_boule}.each do |analyzer|
  p analyzer
  text = <<EOF
  Je mange des carottes, des petits pois et des pèches (bien que ça coule sur les doigts).
  J'aime ça. Un chevalier mange du cheval et élève des chevaux.
  Le peintre utilise un chevalet. Je lis des O'Reilly.
EOF
  a = tire.analyze(text, :analyzer => analyzer)
  print "    "
  p a['tokens'].map{ |t| t['token']}
end

Ce qui donne :

"whitespace"
    ["Je", "mange", "des", "carottes,", "des", "petits", "pois", "et", "des",
    "pèches", "(bien", "que", "ça", "coule", "sur", "les", "doigts).",
    "J'aime", "ça.", "Un", "chevalier", "mange", "du", "cheval", "et",
    "élève", "des", "chevaux.", "Le", "peintre", "utilise", "un", "chevalet.",
    "Je", "lis", "des", "O'Reilly."]
"standard"
    ["je", "mange", "des", "carottes", "des", "petits", "pois", "et", "des",
    "pèches", "bien", "que", "ça", "coule", "sur", "les", "doigts", "j'aime",
    "ça", "un", "chevalier", "mange", "du", "cheval", "et", "élève", "des",
    "chevaux", "le", "peintre", "utilise", "un", "chevalet", "je", "lis",
    "des", "o'reilly"]
"simple"
    ["je", "mange", "des", "carottes", "des", "petits", "pois", "et", "des",
    "pèches", "bien", "que", "ça", "coule", "sur", "les", "doigts", "j",
    "aime", "ça", "un", "chevalier", "mange", "du", "cheval", "et", "élève",
    "des", "chevaux", "le", "peintre", "utilise", "un", "chevalet", "je",
    "lis", "des", "o", "reilly"]
"french"
    ["mang", "carott", "petit", "pois", "pech", "bien", "ça", "coul", "doigt",
    "aim", "ça", "chevali", "mang", "cheval", "élev", "cheval", "peintr",
    "utilis", "chevalet", "lis", "o'reilly"]
"francais"
    ["mange", "carottes", "petits", "pois", "peches", "bien", "que", "coule",
    "sur", "doigts", "aime", "chevalier", "mange", "du", "cheval", "eleve",
    "chevaux", "peintre", "utilise", "chevalet", "lis", "o'reilly"]
"francais_boule"
    ["mang", "carott", "petit", "pois", "pech", "bien", "ça", "coul",
    "doigt", "j'aim", "ça", "chevali", "mang", "cheval", "élev", "cheval",
    "peintr", "utilis", "chevalet", "lis", "o'reilly"]

Le filtre whitespace est clairement là à titre pédagogique, il garde la ponctuation. standard donne un résultat brut mais utilisable, simple apporte une meilleur gestion des apostrophes qui brime les irlandais. french est moins généreux en mots vides que francais, snowball raccourcit bien, mais il lui manque un filtre de stopwords (que l'on peut lui ajouter).

Conclusion

Il n'existe pas de configuration ultime pour Tire... Chaque projet est spécifique, il est donc indispensable de tester et d'affiner les analyseurs avec de vraies données.

In the HTTP world, having real-time applications implies server-push technologies, and for long time doing it properly was painful because not well understood by our browsers.

But with the rise of modern technologies like Websockets and Server-Sent Events, we can build serious HTTP based (soft) real-time applications.

Today, we will see how to use Erlang super powers for the good of the real-time Web. For this, we will write a Server-Sent Event API for Cowboy.

About Server-Sent Events

In a previous blog post, François told you a bit about Server-Sent Events. But for those of you who did not follow so far, here is a little recap.

Server-Sent Events (SSE) is a simple but not well known specification which comes with HTML5. Unlike WebSockets, implementing Server-Sent Events is trivial because you don't need to UPGRADE your request.

The only thing your server have to know is how to stream data. And guess what? Every Erlang HTTP server is capable of that!

Here is an example of what a SSE response looks like:

HTTP1/1 200 OK
Content-Type: text/event-stream

data: an event

data: a new one

As you see, it's a every simple protocol:

• Respond to the request with a text/event-stream Content-Type
• Prefix your streamed events with a data: string

On the client side the javascript API is very simple too:

var source = new EventSource("http://example.com/streaming")

source.onmessage = function(message) {
    console.log("got a new message:", message.data)
}

Which should ouputs in your console:

got a new message: an event
got a new message: a new one

The Cowboy example

So let's start with an Erlang example. For this we will write a Cowboy handler:

-module(eventsource_emitter).
-behaviour(cowboy_http_handler).
-export([init/3, handle/2, terminate/2]).

init({_Any, http}, Req, []) ->
    {ok, Req, undefined}.

handle(Req, State) ->
    Headers = [{'Content-Type', <<"text/event-stream">>}],
    {ok, Req2} = cowboy_http_req:chunked_reply(200, Headers, Req),
    handle_loop(Req2, State).

handle_loop(Req, State) ->
    {ok, Req, State}.

terminate(_Req, _State) ->
    ok.

For the moment the handler does nothing. It's a basic skeleton which responds by a chunked reply and the right Content-Type. You certainly have noticed the handle_loop/2 function which does not belong to the behaviour. This is where the event streaming will occur.

In Cowboy, each request has its own lightweight process. We want this process to wait for Erlang events and send them to the browser. Moreover, we want the process to stop when it receive a shutdown message. So our handle_loop/2 method now looks like the following code:

handle_loop(Req, State) ->
    receive
        shutdown ->
            {ok, Req, State};
        Message ->
            Event = ["data: ", Message, "\n\n"],
            ok = cowboy_http_req:chunk(Event, Req),
            handle_loop(Req, State)
    end.

Ok, so we now have a proper handler which is able to listen for external string events and send them to the browser.

But who will send these Erlang messages to our process? As we are writing an example here, we'll keep it simple and set 2 timers when initializing our handler:

• The first is a recurrent timer which sends our messages
• The second is a simpler timer which sends a shutdown message to stop the streaming.

So we modify the init/3 function according to our specs:

init({_Any, http}, Req, []) ->
    timer:send_interval(1000, "Tick"),
    timer:send_after(10000, shutdown),
    {ok, Req, undefined}.

The handler is now complete. But what our example does exactly?

• First, it sends a Tick message every second to the browser.
• After 10 messages (i.e. 10 seconds) it closes the connection as the process terminates.
• Then on the client side, the browser waits for a few seconds (depending on your browser) and retries a connection. This re-open an event stream and everything happens again.

Here the benefits of Erlang are obvious, because each connection has its own process and is able to accept messages from other processes (another client connection for instance). Think of this as a start for higher applications like pubsubs or queues.

See it in action

This example is part of the cowboy_examples repository, so feel free to run it an see it in action:

$ git clone https://github.com/extend/cowboy_examples
$ cd cowboy_examples
$ make
$ ./start

Then open a browser to http://localhost:8080/eventsource and inspect the javascript source.

Going further

The point of this article was to show you how it's damn simple to implement Server-Sent Events with Erlang.

But don't forget that it's a richer API. Among other:

• Ids: attach ids to your events and detect disconnections
• Type: attach types to your events and listen for these specific ones from the Javascript API
• Retry: modify the interval between reconnections

To dig a little deeper, you can take a look at those links:

• Server-Sent Events specification
• Stream Updates with Server-Sent Events
• em-eventsource - an EventMachine client for Server-Sent Events

An advice often seen in the Rails world is DRY, Don't Repeat Yourself. As explained on the c2 wiki, duplication (whether inadvertent or purposeful) can lead to maintenance nightmares, poor factoring, and logical contradictions, and so, you should try hard to avoid it.

We deploy our projects with Capistrano, which is a good thing to fight the Not Invented Here syndrome. But the capistrano rules were cloned and modified from one project to another, which was a strong sign that we violated the DRY principle.

So I decided to regroup our rules in a gem: capistrano-af83. This ruby gem is an extraction of the rules of our different projects to avoid their duplication. To be honest, there were other goals in this gem, I'll explain them later.

What I discovered at first was that our rules weren't as good as I hoped so, and the duplication took a large part of it. For example, two projects were setting :keep_releases but didn't call the deploy:cleanup task. I'm in favor of having this rule by default in capistrano but it's not the case for the moment, so I put these 2 lines in our default configuration:

set :keep_releases, 5
after "deploy:update", "deploy:cleanup"

Another example of why duplication of capistrano rules is bad: an enhancement of a task on one project is not automatically applied on the other projects. We were having 2 tasks to handle the mongoid.yml file that behave differently. One was better than the other, but the later was used because the guy who wrote it didn't know of the other way! Now, this rule is in capistrano-af83 (mongoid.rb), so it won't happen again.

As I said, I made capistrano-af83 with other aims. First, I wanted to enforce some conventions for naming the environment. We now have a stages.rb to create our 3 default environments: dev, staging and production.

Secondly, it's become easier to use capistrano on a fresh project as we have a template for Rails and I hope to add others templates later.

After using capistrano-af83 on several projects, I can say it's a real improvement for us. But I'd like to know how do you do for your own projects? Do you practice the clone and modify approach? Or do you have a repository of common rules for your projects? Please explain you way and what you feel are the pros and the cons in the comments.

Happy new year!

2012 is here and you can see the new version of our devblog. After Wordpress and Drupal, it is now powered by Jekyll, a static site generator.

Why this change?

The Drupal instance was not really well maintained but the main reason was to have a devblog that looks more like us. We cherish minimalism and carefully chosen features, and not tons of things that someone thought it might be useful but finally, nobody uses. So, even if this version of the devblog still lacks a good theme and some features, it better reflects us and it will be a good base for progressive improvements.

In the last days of 2011, we discussed a lot about our vision of the devblog. The Lean Startup is a trending topic (it deserves its own post but for the moment, you can refer to the wikipedia page for more informations about it), and one of its core principle is the Build-Measure-Learn cycle (again, we should publish a post about it). For the devblog, we formulated 4 hypotheses with a way to measure where we are for each of them:

1. Writers' motivation

In 2011, we were nearly 40 people who had the possibility to write on the devblog. 10 of us wrote at least one post but only 3 were regular contributors. So for 2012 we hope to enlarge our diversity of writers and aim to have 12 writers within the next 6 months, and 6 of them publishing more than one post.

2. Regular publications

In 2011, we wrote 43 posts, so it's about 3.5 posts per month. Not bad, but it means that there were some weeks with no new posts. Our goal is to have 2 posts per week by 15th February.

3. Interested readers

With a regular flow of new posts, it seems logical to have more readers, particularly if the contents are interesting, informative and enlightening. Google analytics says we have about 5,000 page views per month. We want to double this figure: reach 10,000 page views per month by 1st June.

4. Good feedbacks

Our last hypothesis is that if we success on our first 3 goals, our fame will increase. We don't find a metric that satisfies us, so don't hesitate to post a comment if you have a suggestion for us. For the moment, we will count the number of emails we receive that spoke of the devblog. 1 email per week mentioning the devblog by 1st June is our target.

And of course this new devblog is our MVP (Minimum Viable Product). So expect some updates about the progress on our path to these goals, and I hope you'll enjoy reading this devblog in 2012.

We are thinking about using gitlabhq for some of our internal repositories.

But then it would be cool to be able to do some full text searches on the code as well as on commit messages, tags etc..

So I wrote a quick and very dirty proof of concept (Ruby) to search a git repository through elastic search (which in its own right is extremely cool, and we should write a big article about it soon).

• uses the grit and tire gems, please notice that grit is self hosted on github

Note: as per default Grit behaviour it gets the current master but its trivial to have it index other stuff.

Note to self: Could be useful with gitlabhq

Installation

bundle install

You should have a local elastic search running.

Usage:

# Inits with a full path to a local git repository
repo_index = GitElasticSearch.new("/Users/git/repositories/gitlabhq.git")  
# Inits elastic search index       
repo_index.init!
# Indexes the repo                 
repo_index.index!

To search the code:

Tire.search repo_index.index_name do
   query do
     string 'mysearchterm'
   end             
end

You can look at example.rb to see the full example

Todo

Well basically everything, but let's start with writing some tests and then different branches, adding the commit messages, make elastic search configurable and other fun stuff

Please remember : this is a very quick and a very dirty implementation.

Licence

BSD Copyright Ori Pekelman, AF83 2011

Troisième et dernière semaine d'ActiveDaily, une astuce quotidienne sur ActiveSupport et ActiveModel.

Vous pouvez suivre l'ActiveDaily sur github ou bien lire ces billets de blog hebdomadaire.

Cette semaine on se concentre sur ActiveModel, alors on installe un nouveau joyau:

gem install activemodel

Et puis on essaie de l'inclure.

require 'active_model' # Un underscore mais pas de /all

ActiveModel est la base d'ActiveRecord mais aussi de mongoid, tire, datamapper, redis_orm, ...

ActiveModel::Callbacks

Si les callbacks d'ActiveSupport sont un peu rude, ceux d'ActiveModel sont plus sympa.

require 'active_model'

class Bim
  extend ActiveModel::Callbacks
  # On déclare les callbacks disponibles
  define_model_callbacks :kick

  # L'implémentation
  def kick
    run_callbacks :kick do
      puts 'BIMMMMMMMMM!'
    end
  end
end

class Norris < Bim
  # on rajoute les callbacks
  before_kick :before_kick
  after_kick :after_kick
  # on peut aussi utiliser around_kick :ma_methode

  def before_kick
    puts 'Be careful'
  end

  def after_kick
    puts 'KO'
  end
end

chuck = Norris.new
chuck.kick # Your are KO baby

Alors n'hésitez pas a rajouter des callbacks dans vos classes.

http://rubydoc.info/gems/activemodel/3.1.1/ActiveModel/Callbacks

ActiveModel::Observer

Les observers permettent de réagir à des actions d'une autre classe. Cela permet de découpler facilement deux composants. Dans la class Observé, il faut simplement appelé notify_observers.

require 'active_model'

class Norris
  include ActiveModel::Observing

  def kick
    notify_observers(:before_kick)
    puts 'BIMMMMMMMMM!'
    notify_observers(:after_kick)
  end
end

class NorrisObserver < ActiveModel::Observer
  def before_kick(norris)
     puts "start the video recording"
  end

  def after_kick(norris)
    puts "switch off the video recording"
  end
end

Norris.observers = NorrisObserver
Norris.instantiate_observers
chuck = Norris.new
chuck.kick

On arrive rapidement a avoir un comportement similaire. Seule petite déconvenue, je pensais que ActiveModel::Observing se branchait sur les callbacks.

http://rubydoc.info/gems/activemodel/3.1.1/ActiveModel/Observer http://rubydoc.info/gems/activemodel/3.1.1/ActiveModel/Observing/ClassMethods

ActiveModel::MassAssignmentSecurity

MassAssignmentSecurity permet de protéger l'écriture de certains attributs en fonction du rôle. MassAssignmentSecurity fournit deux facons de se protéger. Avec attr_accessible en mode whitelist ou avec attr_protected en mode blacklist.

require 'active_model'

class Norris
  include ActiveModel::MassAssignmentSecurity

  attr_accessible :name
  attr_accessible :name, :guns, :as => :chuck_norris

  def update(params, role=:default)
    sanitize_for_mass_assignment(params, role)
  end
end

chuck = Norris.new
p chuck.update({name: 'chuck', guns: 2}) # {name: 'chuck'}
p chuck.update({name: 'chuck', guns: 2}, :chuck_norris) # {name: 'chuck', guns: 2}

http://rubydoc.info/gems/activemodel/3.1.1/ActiveModel/MassAssignmentSecurity/Cla

ActiveModel::Dirty

Dirty vous aide a tracer les changements dans un objet.

require 'active_model'

class Norris
  include ActiveModel::Dirty # Chuck Norris cannot be dirty, should throw an exception

  define_attribute_methods [:first_name]
  attr_reader :first_name

  def first_name=(val)
    first_name_will_change! unless val == @first_name
    @first_name = val
  end
end

chuck = Norris.new
chuck.first_name = 'plop'
chuck.changes
chuck.changed_attributes

Vous avez ainsi accès à quelques méthodes:

chuck.changed? # true
chuck.first_name_changed? # true
chuck.first_name_was # nil

http://rubydoc.info/gems/activemodel/3.1.1/ActiveModel/Dirty

C'est le dernier ActiveDaily. Je préfère m'arrêter sur un numéro 13.

Vous avez raté:

• des choses dans ActiveModel::AttributeMethods utilisé entre autre par ActiveModel::Dirty
• ActiveSupport::Concern qui facilite l'écriture de mixins.
• ActiveSupport::Inflector qui permet de transformer des mots du singulier au pluriel et d'autres truc aussi
• Hash#except, l'inverse de Hash#slice
• ActiveSupport::Notifications avec lequel je n'ai pas encore joué

Deuxième semaine d'ActiveDaily, une astuce quotidienne sur ActiveSupport et ActiveModel.

Vous pouvez suivre l'ActiveDaily sur github ou bien lire ces billets de blog hebdomadaire.

ActiveSupport::Callbacks

Les callbacks permettent de facilement rajouter des points d'entrées dans vos classes.

require 'active_support/callbacks.rb'

class Bim
  include ActiveSupport::Callbacks
  define_callbacks :kick

  def kick
    run_callbacks :kick do
      puts 'BIMMMMMMMMM!'
    end
  end
end

class Norris < Bim
  set_callback :kick, :before, do |chuck|
    puts 'Be careful'
  end

  set_callback :kick, :after do |chuck|
    puts 'KO'
  end
end

chuck = Norris.new
chuck.kick # Your are KO baby

Bon dans Rails toute cette logique est utilisé dans ActiveModel, rendez vous la semaine prochaine :).

ActiveSupport::OrderedOptions

OrderedOptions vous permet de setter des paramètres de config d'une manière assez facile.

require 'active_support/ordered_options'

options = ActiveSupport::OrderedOptions.new
options.chuck = 'norris'
options[:chuck] # norris

Les options dans Rails sont gérès de cette manière.

Hash#slice

Il peut être pratique de filtrer les clés d'un Hash.

require 'active_support/core_ext/hash/slice'

hash = Hash.new
hash['chuck'] = 'norris'
hash['bruce'] = 'lee'
hash['roger'] = 'hanin' # my real idol
hash.slice('bruce', 'roger')

D'ailleurs vous pouvez le combinez avec le HashWithIndifferentAccess.

hash.with_indifferent_access.slice(:bruce, :roger)

Object#try

Si vous en avez assez de tester sans cesse si la variable que vous avez est nil, vous pouvez utiliser try.

require 'active_support/core_ext/object'

a = nil
a.try(:size) # nil
b = "plop"
b.try(:size) # 4

Vous avez une astuce sous le coude que vous désirez partager ? Envoyez la moi et elle se retrouvera peut être dans un prochain ActiveDaily. Seule les astuces sur ActiveSupport et ActiveModel seront incluse (sauf si vous arrivez a me convaincre).

Dans la lignée du GitDaily, il y a désormais l'ActiveDaily, un voyage dans le monde d'ActiveSupport et d'ActiveModel.

Vous pouvez suivre le ActiveDaily sur github ou bien lire ces billets de blog hebdomadaire.

Les exemples ont été testé avec ruby 1.9.2.

Pour bien démarrer :

gem install activesupport i18n # Oui il faut i18n pour que activesupport fonctionne a peu près correctement
gem install pry # pour le plaisir

Pour exécuter les exemples :

pry # ou irb
require 'active_support/all' # oui un underscore :(

Blank?

Pour savoir si un objet est vide. C'est à dire soit false, empty, contenant seulement des espaces, ...

 require 'active_support/all' # violent mais efficace, sinon require 'active_support/core_ext/object'

nil.blank? # true
"".blank? # true
"\n".blank? # true
false.blank? # true
[].blank? # true
{}.blank? # true
true.blank? # false
"Chuck".blank? # false

ActiveSupport::Duration

ActiveSupport Duration permet de manipuler d'une manière plutôt sympathique le temps.

require 'active_support/all' # violent mais efficace, sinon require 'active_support/core_ext/object' et require 'active_support/duration'

1.month.ago # months, days/day, hour/hour ...
1.month.since(2.hours.ago) # Calcul simple
(1.month + 1.day).ago

Le calcul simple ou la configuration de date devient plus lisible.

Devise l'utilise par exemple pour configurer le délai maximum de confirmation de l'email:

config.confirm_within = 2.days

HashWithIndifferentAccess

Les HashWithIndifferentAccess permettent de manipuler les clés indifféremment avec un symbole ou la clé originale.

require 'active_support/hash_with_indifferent_access'

h = HashWithIndifferentAccess.new # {}
h[:chuck] = "Norris" # "Norris"
h[:chuck] # "Norris"
h["chuck"] # "Norris"

On me dit que c'est utilisé entre autre pour le hash params dans Rails.

ActiveSupport::MessageVerifier

Pour générer des tokens d'accès et vérifier leur authenticité, MessageVerifier est fait pour vous.

require 'active_support/message_verifier'

v = ActiveSupport::MessageVerifier.new("houhou this is a secret")
token = v.generate(["chuck", "norris"]) # you get an unreadable string
chuck, norris = v.verify(token)

Dans le cas d'un token qui a été modifié, la signature devient invalide, une exception est alors jeté.

ActiveSupport::MessageVerifier::InvalidSignature

Dans un contexte Rails, vous pouvez utiliser le secret_token de l'application :

ActiveSupport::MessageVerifier.new(Rails.configuration.secret_token)

Si vous devez encore plus protéger les données, vous pouvez utiliser MessageEncryptor qui dispose d'une API similaire.

ActiveSupport::StringInquirer

Vous avez envie de poser des questions à vos chaines de caractères ? StringInquirer est fait pour cela.

require 'active_support/all'

chuck = "norris".inquiry # équivalent a ActiveSupport::StringInquirer.new("norris")
chuck.norris? # true
chuck.other? # false

Vous savez désormais comment fonctionne l'interrogation de la production?.

Vous avez une astuce sous le coude que vous désirez partager ? Envoyez la moi et elle se retrouvera peut être dans un prochain ActiveDaily. Seule les astuces sur ActiveSupport et ActiveModel seront incluse (sauf si vous arrivez a me convaincre).

It is not always easy to do complicated queries over ETS tables or mnesia.

The official syntax to make queries with select/2 is really cryptic.

Here is an example from the official Erlang docs. The query returns the name of each male person aged more then 30.

MatchHead = #person{name='$1', sex=male, age='$2', _='_'},
Guard = {'>', '$2', 30},
Result = '$1',
mnesia:select(Tab,[{MatchHead, [Guard], [Result]}]),

Criterias are expressed with $ and the whole thing becomes quite convulted for anything more complicated.

Furthermore it is impossible to do what would be basic operations in other database engines, like sorting the results.

But, a module exists that makes queries better legible . QLC that stands for Query List Comprehension. It supports Mnesia, ETS and DETS.

Here is the previous query, rewritten:

Query = qlc:q([Person#person.name || Person <- mnesia:table(Tab), Person#person.sex == male, Person#person.age > 30]),

In this case the query is expressed as a list comprehension. Criterias are written in a comprehensible manner in the second par of the list comprehension.

If you want to execute this query in Mnesia, you have to do so in a transaction.

-include_lib("stdlib/include/qlc.hrl")
Transaction = fun() ->
    Query = qlc:q([Person#person.name || Person <- mnesia:table(Tab), Person#person.sex == male, Person#person.age > 30]),
    qlc:eval(Query)
end,
mnesia:transaction(Transaction),

To efficiently sort the result, qlc provides qlc:sort.

-include_lib("stdlib/include/qlc.hrl")
Transaction = fun() ->
    Query = qlc:q([Person#person.name || Person <- mnesia:table(Tab), Person#person.sex == male, Person#person.age > 30]),
    Order = fun(A, B) ->
        B#person.age > A#person.age
    end,
    qlc:eval(qlc:sort(Query, [order, Order]))
end,
mnesia:transaction(Transaction),

I have not yet explored all the possibilities of QLC but the syntax it offers has convinced me to use it as often as posible. The query maintainance becomes much easier.

One last thing. It is possible to create your own modules that can be used with QLC. You can propose a search syntax that is much nicer with a backend altogether different from ETS or Mnesia.

Chez af83, nous avons pour tradition de faire des présentations les lundis matin et/ou des ateliers pratiques. Et ce coup-ci, j'ai décidé de présenter Dart et de donner mon avis sur ce nouveau langage.

Google souhaite remplacer le JavaScript par un langage plus performant, bénéficiant d'une meilleure intégration avec les outils (IDE, intégration continue, etc.) et qui n'aurait pas les erreurs de jeunesse du JavaScript. Les ingénieurs de Google ont donc proposé un nouveau langage, Dart, très inspiré de Java.

Je suis déçu par ce langage qui ne présente pas grand chose d'innovant (à part les Isolates éventuellement) et qui semble surtout attrayant pour les développeurs de Java et C++ qui n'apprécient pas le JavaScript. Et je vois mal Dart tuer le JavaScript pour la simple et bonne raison que Microsoft et Mozilla ne voudront pas l'implémenter dans leurs navigateurs.

Les slides sont visibles sur http://nono.github.com/Presentations/20111024_Dart/