Planet Smalltalk

February 06, 2016

Pharo Weekly - More Enhancements

17526 growMemoryByAtLeast: duplicated

17530 fix testDangerousClassesConditions

17242 String>>compare:caseSensitive seems to be failing for extended charset comparisons
17521 recategorizing uncategorised method

17522 Failing test: ReleaseTest>>#testInstalledMethodsWithIncorrectTrailerKind

17479 Interval class comment mixes step/stop when explaining Instance Variables
17366 wrong TempsNamesQCompress method trailer ?

17406 Renaming a class using the class menu “rename” option allows for non-capitalized class name
17516 add #isFromSharedPool to OCLItteralVariable

17495 Decompiler cant decompile Object>>#instVarAt: due to error code

17508 Deprecate Unused Morph methods: submorphNamed: and submorphNamed: aName ifNone: aBlock
17515 fix minimal Pharo shrinking build

17512 Spec should use FastTable

17501 ProcessTest>>testHighPriorityOverridesWaitTime intermittent failures

17035 The smallSaveIcon is the same as the smallSaveAsIcon

17513 ThemeIcons>>#downloadFromUrl not working

17500 Implement class creation methods for new layouts

17491 find selector shows syntax error debugger
16735 Replace Startup/Shutdown list with SessionManager

17490 Command Line Handler test runner should print a small stack for failures and errors

17496 add table for all defined method selectors and use it in code completion and syntax highlighting

17466 testVisitXMLParserConfigurationWithVersion101 failing

17438 New Rubric Configuration

17454 QualityAssistant v2.3.3
17474 adapt Athens to UnifiedFFI 0.11 (pointer arity logic)

17421 RubTextMethodLink and RubTextClassLink Leaks (TextStyling related)

17442 Improve Morph>>#flash

17348 Enhance shortcut learnability

17452 UIManager has a test method …

17022 Return sequence not always emitted
17296 Remove Delay>>otherwise:

17453 add #genForRBArgumentNode
17423 PopupChoiceDialogWindow keyboard navigation

17435 ClassOrganisation>>#changeFromString: MNU for empty change
17301 Morph>>#drawOnAthensCanvas: should only fill innerBounds

17384 RBNulllFormater>>#format: should not return nil

17404 ExternalBrowser can not show meta class definition

17443 RubTextAreaExamples>>#show:while: should not use #displayWorldSafely

17457 Change Growl setting name to PopupNotifier
17444 SVColorSelectorMorph should not use (World displayWorldSafely)

17408 The generic stack debugger defines double meaning for some keybindings when editing code

17335 We should add a #ffiSizeOf:

17267 fix senders of #compactClassesArray

17077 ReleaseTest testMethodsWithUnboundGlobals fails for some slotexamples

17425 unload BaselineOfFFINB

17427 deprecated Form class>>#unload

17415 SimpleButtonMorph lacks #themeChanged

17357 Lost instance variables (AthensCairoPatternSurfacePaint)

17422 Rename FFI-NB to UnifiedFFI

17419 Rename AthensCairoCanvas setAlpha: to drawWithAlpha:

17416 Integrate GTools version 3.9

Pierce Ng - Metaprogramming Pascal with Mustache in Smalltalk

A while back I wrote about command line scripting of Pharo to backup an SQLite database, specifically, this server’s iptables log. Eventually I wrote a program in Pascal to archive and rollover the log database. Reason being, using the excellent Free Pascal compiler, I link SQLite statically into the final executable, meaning I only deploy a single binary executable, which is a tad more convenient than deploying the Pharo VM, image, sources, and script files.

Well, not quite just one file.

My program, iptlb, uses the SQLite online backup API to back up the running log database to an archive database file, then “backs up” an empty template database to the running log database file, effectively overwriting it. Because I want the flexibility to change the database schema during each backup run, iptlb creates a new template database each time it is invoked. Because I also want just one Pascal source file, I want to store the database schema in the source file itself, so that the schema gets version-controlled along with the Pascal code.

However, Pascal does not support multi-line string literals:

(* This Pascal code is invalid. *)
  dbSchema = '
    create table x (xk int, xv varchar);
    create table y (yk int, yv int);

This means that I cannot embed the database schema into iptlb’s Pascal source directly as a multi-line string. In production, I have to also deploy the database schema SQL as a separate file which is read by iptlb to create its template database.

There is a workaround in Pascal for lack of multi-line strings, as follows:

  sl: TStringList;
  dbSchema: String;
  sl := TStringList.create;
  sl.add('create table x (xk int, xv varchar);');
  sl.add('create table y (yk int, yv int);');
  dbSchema := sl.text; // dbSchema is now the multiline string.;

This looks like a templating thing. And that calls for Mustache!

After fiddling around, I created a simple class PMMultiLineStringTemplate which wraps the work with Mustache. Here’s the motivating use case:

| tmpls mst |

"The Mustache template of the Pascal code."
tmpls := '
function {{functionName}}: String;
  sl: TStringList;
  sl := TStringList.create;{{#stringList}}
  {{functionName}} := sl.text;;

mst := PMMultilineStringTemplate new.
mst atAttribute: 'functionName' putValue: 'dbSchema'.
mst template: tmpls.

FileSystem disk workingDirectory / 'ulog-schema.sql' readStreamDo: [ :rs |
    rs ascii.
    [ rs atEnd ] whileFalse: [
        | x |
        x := rs nextLine.
        (x size > 0) ifTrue: [
            mst atAttribute: 'stringList' withSubKey: 'sline' putValue: x ]]].

FileSystem disk workingDirectory / 'dbschema.pas' writeStreamDo: [ :ws |
    ws ascii;
        lineEndConvention: #lf; 
        nextPutAll: mst value ]

Copying the output of dbschema.pas (generated with a proper schema as above) into iptlb.pas, I now have one source file iptlb.pas embedding the SQL schema, and I deploy one binary executable iptlb only.

February 05, 2016

Torsten Bergmann - Aida 6.6 runs on newest Squeak 5

The Weekly Squeak - Aida 6.6 runs on newest Squeak 5


Janko writes:

Dear all,

Squeak 5 together with a shiny new website was released
this summer so it was a time to prepare Aida for this release as well. On you can find a link to All-in-One package to
run Aida 6.6 on latest Squeak 5 for Linux, Mac and Win.

Thanks to Tobias Pape for help and Robert Hirschfeld for inspiration!

Best regards

Torsten Bergmann - Large file uploads in Seaside

Nice article on how to use Seaside and Nginx to handle large file uploads

Torsten Bergmann - Pharo MOOC available

The registration for Pharo Massive Open Online Course (Pharo MOOC) is available

Torsten Bergmann - IPFS

SmallIPFS - Smalltalk Interplanetare Filesystem API. Read more and have a look at the project page

Joachim Tuchel - MOOC on Live Object Programming in Pharo

Ever wondered why Smalltalk developers love their environment and what all the buzz about Live Object Programming is about? Would you like to learn Smalltalk and understand why Smalltalkers are so passionate about it? Maybe this Free online course is the right opportunity for you to start discovering a new level of programming and widen […]

Johan Brichau - Large file uploads in Seaside

Uploading files in a Seaside web application is easy. Unfortunately, there is a drawback to the easiness: entire files are loaded into the Seaside backend’s memory because an instance of WAFile contains the entire file’s contents. In many situations, loading the file contents in the Seaside backend’s memory is not necessary (for example, if the file only needs to be stored on disk) or even impossible (e.g. in the case of extremely large files).

This post details an extension to Seaside that works together with the NGINX front-end web server and its file upload module. The uploaded file is stored to disk and NGINX subsequently only passes a file reference to the Seaside backend. This off-loads the heavy lifting to the web server and prevents memory overload in the Seaside backend while still keeping close to the ease of implementation of “traditional” file uploads in Seaside.

Many of you will notice that this solution is based on work by Nick Ager, whose blog post has unfortunately disappeared from the web. Since there have been several questions on the Seaside mailinglist on this topic, I thought it would a good idea to revisit our implementation (which has been working in production for years now) and make it usable as a separate Seaside extension.


The Seaside extension to support large file uploads is in the optional Seaside-ExternalFileUpload package. At the time of writing of this post, you need to load this package manually in your Seaside3.x image. The package should work well in 3.0, 3.1 and 3.2.

NGINX with file upload module

You need to compile NGINX from source because, like many of its modules, the file upload module is not included in the binary distributions. Also, since NGINX 1.3.0 or so, you need to make sure to use version 2.2 of the file upload module. Executing the following commands should work for you, but mind you might need to pass additional configuration options to the configure command to fit your NGINX setup.

  sudo curl -O
  sudo tar xf nginx-1.8.1.tar.gz
  cd nginx-1.8.1
  sudo curl -L -o nginx-upload-module-2.2.0.tar.gz
  sudo tar xf nginx-upload-module-2.2.0.tar.gz
  sudo mv vkholodkov-nginx-upload-module-aba1e3f nginx-upload-module-2.2.0
  sudo ./configure --add-module=./nginx-upload-module-2.2.0/
  sudo make install

NGINX configuration

Once you get NGINX installed, you need to configure an upload location in the server block that concerns your Seaside app. The following configuration defines that location as the path /fileupload, which means that the file upload plugin is listening at that location. Files uploaded to that location will be stored in the upload_store directory on the server. In our case, the files will be uploaded to /var/www/uploadstore.

Once the file is uploaded, the request that is sent to the Seaside back end (listening at location /) has the additional fields name, content_type and path that contain the respective properties of the uploaded file. These properties will be available in the Seaside callback attached to the file upload field. Finally, the configuration also ensures all (other) fields of the form are sent to Seaside such that all callbacks of the form are executed there.

Please see the file upload module documentation for more information on these and other configuration parameters.

# Upload form should be submitted to this location
location ~ /fileupload {

  # Pass altered request body to this location
  upload_pass /;

  error_page 405 415 = /;

  # Store files to this directory
  upload_store /var/www/uploadstore;

  # Allow uploaded files to be read only by user
  upload_store_access user:rw group:rw all:rw;

  # Set specified fields in request body
  upload_set_form_field $upload_field_name "";
  upload_set_form_field $ "$upload_file_name";
  upload_set_form_field $upload_field_name.content_type "$upload_content_type";
  upload_set_form_field $upload_field_name.path "$upload_tmp_path";

  # seaside automatically assigns sequential integers to fields with callbacks
  # we want to pass those fields to the backend
  upload_pass_form_field "^\d+$";

  upload_cleanup 400 404 499 500-505;

Example File Upload

The package Seaside-ExternalFileUpload contains an example component WAFileUploadExample that demonstrates “traditional” file uploads side-by-side with the new “external” file uploads. Here is the snippet for such an external file upload (i.e. where the upload is handled by the front-end web server NGINX):

html form
  fileUploadLocation: 'fileupload';
  with: [
      html externalFileUpload
          callback: [ :ef | file := ef ].
      html submitButton
          text: 'Upload file via front-end'

Like any form with a file upload field, you need to set it to be multipart. Next, you need to pass the fileUploadLocation, which is the location configured in NGINX to handle file uploads. In our case, this is fileupload, but you can choose any name you want for that location as long as you use the same name here and in the NGINX configuration. The file upload field tag is externalFileUpload (instead of fileUpload). The callback block of this field [ :ef | file := ef ] will be invoked with a WAExternalFile instance instead of a WAFile instance. A WAExternalFile contains the uploaded file’s filename, its content type and path on disk. From here on, it’s up to you what to do with the file. A good idea is to move the file to its proper location, for example.

The Seaside-ExternalFileUpload package is currently a preview package. It will evolve as we integrate it further, for example by adding support for the ajax file uploads (as they are part of Seaside 3.2) and the jQuery file upload plugin. More about this in upcoming posts.

Please contact us on the Seaside mailinglist in case you need help or for any additional comments and remarks.

February 04, 2016

Pharo Weekly - An example of debugging with moldable tools


One thing we noticed over the past year since we introduced GT in Pharo is that people still tend to use these tools in a similar way classic tools were used. We think there is a more potential in these tools.

To change this, we would like to collect stories of how using these tools enabled a workflow that was otherwise not possible. To get this started, we will start to document more consistently some of the sessions, and we would like to get input from your as well. The idea is to create a catalog of tutorials that people can follow and get inspired from.

Let me start. I recently had to debug a small problem, and I ended up having an experience that I found beautiful:

What do you think?


Benoit St-Jean - Quelques moqueries!

Non, il ne s’agit pas d’un billet humoristique! Mais dans ce cas-ci, moquerie se veut la traduction imprécise (à dessein) du terme mock, qui signifie plutôt simulacre dans le cas qui nous occupe.

Un mock, c’est un outil comme un autre.  Même si parfois j’ai l’impression qu’on surévalue grandement son utilité.  En 20 ans de Smalltalk, je n’ai eu qu’à travailler avec des mocks seulement 2 fois!

Plusieurs solutions existent pour votre environnement de développement favori!


Disponible pour Squeak, Pharo, aussi la version pour VisualWorks dans le Cincom Public Repository


Disponible pour Dolphin,et Pharo.


Une version pour VisualAge,


Une version pour Pharo.


Disponible pour VisualWorks et Squeak.


Disponible pour VisualWorks.

Classé dans:Dolphin, Pharo, Smalltalk, Squeak, VisualAge, VisualWorks Tagged: BabyMock, Dolphin, DoubleAgent, MiniSMock, mock, Mocketry, Pharo, SMock, Squeak, Teachable, VisualAge, VisualWorks

February 03, 2016

Benoit St-Jean - Smalltalk en vrac (22)


L’immutabilité dans la VM Cog : enfin!


Un extracteur de headers (les header files en C) pour faciliter vos projets avec FFI.  Les détails ici et ici.

Un article en français dans Linux Magazine.

Classé dans:Cog, Machine virtuelle, Pharo, Smalltalk, VM Tagged: C extracteur, Cog, FFI, header, header files, immutabilité, Linux, magazine, VM

February 01, 2016

Pharo News - Press article about Pharo

<p>New article about Pharo in the issue 190 of the french computer magazine <a href="">GNU Linux Magazine</a>. Learn how to create a PDF file with Pharo and Artefact. </p> <p><figure><img src="/files/posts/gnulinux-magazine-190.jpg"></img><figcaption></figcaption></figure></p>

Pharo News - Press article about Pharo

<p>New article about Pharo in the issue 190 of the french computer magazine <a href="">GNU Linux Magazine</a>. Learn how to create a PDF file with Pharo and Artefact. </p> <p><figure><img src="/files/posts/gnulinux-magazine-190.jpg"></img><figcaption></figcaption></figure></p>

Pharo Weekly - Parsing User-Agent strings with a web service

I want to share a snippet of code.

In HTTP, a client identifies itself using the User-Agent header. That string is long and cryptic. To make sense of it you have to parse it. There are web services that can do this. Here is how you can invoke one of them.

ZnClient new
  host: '';
  queryAt: 'uas' put: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/601.4.4 (KHTML, like Gecko) Version/9.0.3 Safari/601.4.4';
  queryAt: 'getJSON' put: 'all';
  contentReader: [ :entity | NeoJSONReader fromString: entity contents ];

Since STON parsing is backward compatible with JSON, you can use that as well.

ZnClient new
  host: '';
  queryAt: 'uas' put: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/601.4.4 (KHTML, like Gecko) Version/9.0.3 Safari/601.4.4';
  queryAt: 'getJSON' put: 'all';
  contentReader: [ :entity | STON fromString: entity contents ];

You will get back a nice dictionary with much more sensible key/value pairs.

Note that this is a public service. I think it would be polite and more efficient if you put a cache in front of it, like LRUCache.


January 30, 2016

Pharo Weekly - Fuel faster and BMP reader :)

Fuel reading is 60% faster than BMP:
Smalltalk garbageCollect; garbageCollect.
r1 := [FileLocator imageDirectory / ‘test.bmp’ readStreamDo: [ :s |
s binary.
fromBMP := BMPReadWriter formFromStream: s]] benchFor: 2 seconds.
Smalltalk garbageCollect; garbageCollect.
r2:=[FileLocator imageDirectory / ‘test.fuel’ readStreamDo: [ :s |
s binary.
fromFuel := (FLMaterializer newDefault materializeFrom: s) root]] benchFor: 2 seconds.
{ r1. r2 } “an Array(a BenchmarkResult(2,199 iterations in 2 seconds 2 milliseconds. 1,098 per second) a BenchmarkResult(5,728 iterations in 2 seconds 3 milliseconds. 2,860 per second))”

January 29, 2016

Pharo Weekly - Travis CI integration added to OSSubprocess and FFICHeaderExtractor

Hi guys,

This is just to let you know that with the help of Esteban and between some work together [1] [2] I was able to have Travis CI integrated with OSSubprocess and FFICHeaderExtractor [3] [4] . Both projects are built and tested under Linux and OSX. Note also that one of the projects does a lot of FFI calls to system libs like libc and the other even generates C programs, compiles them and run it. Even the OS dependencies (like installing the compiler) are resolved correctly :)

January 27, 2016

Pharo Weekly - FFI C Header Extractor

Dear all,

I am happy to announce the first release of FFICHeaderExtractor project. You can find the project and the documentation here:

Pharo News - [ANN] Pharo on Bountysource

<p>We have set up a Bountysource Team for Pharo. This means that everyone can suggest bounties, contribute money to existing bounties or support Pharo development directly with a monthly contribution.</p> <p>Have a look here: <a href=""></a></p> <p>Existing bounties can be found here: <a href=""></a></p> <p>To learn more about bounty source, see the <a href="">FAQ</a></p> <p>For supporting Pharo development with a monthly contribution, see </p> <ul><li> <a href=""></a></li></ul>

Pharo News - [ANN] Pharo on Bountysource

<p>We have set up a Bountysource Team for Pharo. This means that everyone can suggest bounties, contribute money to existing bounties or support Pharo development directly with a monthly contribution.</p> <p>Have a look here: <a href=""></a></p> <p>Existing bounties can be found here: <a href=""></a></p> <p>To learn more about bounty source, see the <a href="">FAQ</a></p> <p>For supporting Pharo development with a monthly contribution, see </p> <ul><li> <a href=""></a></li></ul>

January 26, 2016

Benoit St-Jean - Numerical Methods 2016

La toute dernière version de Numerical Methods est arrivée! La présente édition est une version abrégée où tout le code Java a été enlevé pour ne garder que le plus intéressant : le code Smalltalk.

Maintenue par Stéphane Ducasse et Serge Stinckwich, cette version demeure toutefois fidèle à l’originale de Didier H. Besset.

Pour obtenir cette librairie (SciSmalltalk), vous n’avez qu’à venir ici. Pour être informé des derniers développements, il y a un groupe Google.

Classé dans:Smalltalk Tagged: Didier H. Besset, Numerical Methods, SciSmalltalk, Serge Stinckwich, Stéphane Ducasse

Pharo Weekly - FFICHeaderExtractor first milestone

Hi guys,

OK, I have a first working version and so I wanted to share it with you.
I have not yet the time to start writing the doc since I just finished the first pass on the code. Tomorrow I will start with the doc. But I thought some of you may be interested in taking a look even without formal “doc” (and some feedback/iteration may avoid re-writing docs..).
If you have no clue what I am talking about, then this summary is for you:
When we  use FFI  to call a certain library it’s quite common that we need to pass as argument certain constants (for example, SIGKILL to kill()). These constants are defined in C header files and can even change it’s value in different paltforms. 
These constants also are sometimes defined by the C preprocessor and so there is not way to get those values from FFI. If you don’t have the value of those constants, you cannot make the FFI call. 
I have tested the tool in OSX and CentOS using latest Pharo 5.0. It won’t work in Windows right now.  As usual, all classes and methods have comments and there are enough tests.
At the end, I decided the C program will output a very naive Smalltalk literal array kind of thingy. The tool then parses that output and directly creates a init method (which is compiled into the SharedPool class) for that platform which is then called automatically at startup (only if initialization is needed).
As for real examples, I started to write constants for libc:  signal.h (to use kill()) , wait.h (to use wait() famility), fcntl.h (to use … xxx()) , and errno.h. You can take a look to the package ‘FFICHeaderExtractor-LibC’.
Note that for running the tests you need ‘cc’ findable by path in OSX and ‘gcc’ in Unix.
To load the code in a latest Pharo 5.0, execute:
Metacello new
    baseline: ‘FFICHeaderExtractor’;
    repository: ‘github://marianopeck/FFICHeaderExtractor:master/repository’;
Any feedback is appreciated. 
I will start writing the doc now.
BTW: Big thanks to Eliot Miranda which helped me answering noob questions and providing useful code and guidelines. 

Yoshiki Ohshima - [Person] Marvin Minsky (1927-2016)

I once had a short chat with Marvin. At our Glendale office several years ago, he was taking a break from his meeting and ventured into our office area. I was playing with a puzzle (see pictures above) and told him that there are to solutions for this p ...

Yoshiki Ohshima - [その他] Marvin Minsky (1927-2016)

Marvinがとあるミーティングに参加するためにApplied Mindsのオフィスに来ていて、休憩を取るためにViewpointsのエリアでちょっと時間をつぶしていたことがあります。私はごく簡単な知恵の輪で遊んでいたのですが、実はそのパズルにはふた通りの解があるということに気がついて(写真のように、頭が同じほうを向いている場合と反対を向いている場合)、Marvinにそれを見せつつ、「このパズルにはふた通りのやりかたがあるんだよね。上向きと下向きで電子のスピンみたい」と言ってみたのです。 すると彼 ...

January 25, 2016

Stefan Marr - Towards Meta-Level Engineering and Tooling for Complex Concurrent Systems

Last December, we got a research project proposal accepted for a collaboration between the Software Languages Lab in Brussels and the Institute for System Software here in Linz. Together, we will be working on tooling for complex concurrent systems. And with that I mean systems that use multiple concurrency models in combination to solve different problems, each with the appropriate abstraction. I have been working on these issues already for a while. Some pointers are available here in an earlier post: Why Is Concurrent Programming Hard? ��And What Can We Do about It?

End of February, I am going to talk about that a little more at the Arbeitstagung Programmiersprachen in Vienna. Below, you can find an abstract and link to the position paper. There is not a lot of concrete material in yet, but it sketches the problems we will try to address in the years to come.


With the widespread use of multicore processors, software becomes more and more diverse in its use of parallel computing resources. To address all application requirements, each with the appropriate abstraction, developers mix and match various concurrency abstractions made available to them via libraries and frameworks. Unfortunately, today’s tools such as debuggers and profilers do not support the diversity of these abstractions. Instead of enabling developers to reason about the high-level programming concepts, they used to express their programs, the tools work only on the library’s implementation level. While this is a common problem also for other libraries and frameworks, the complexity of concurrency exacerbates the issue further, and reasoning on the higher levels of the concurrency abstractions is essential to manage the associated complexity.

In this position paper, we identify open research issues and propose to build tools based on a common meta-level interface to enable developers to reasons about their programs based on the high-level concepts they used to implement them.

  • Towards Meta-Level Engineering and Tooling for Complex Concurrent Systems; Stefan Marr, Elisa Gonzalez Boix, Hanspeter Mössenböck; in ‘Proceedings of the 9th Arbeitstagung Programmiersprachen’ (ATPS’ 16).
  • Paper: PDF, HTML
  • BibTex: BibSonomy

January 24, 2016

Smalltalk Jobs - Smalltalk Jobs -1/24/16

  • Los Angeles, CASoftware Developer With Smalltalk through HCL Global Systems
    • Required Skills:
      • A minimum of 7 years of hands on development experience in Object oriented programming.
      • A minimum of 5 years experience with Smalltalk language
      • C++
      • C#
      • Well versed with DB2 table access procedures and SQLs.
      • Should be able to work independently with minimal guidance.
      • Be able to perform reverse engineering for areas in the application where there is no documentation.
      • Be able to perform analyst role when needed.
      • Must have strong communication skills both written and verbal and interact with various departments.
      • Be able to lead certain tasks independently.
      • Self-motivated and a good team player.
    • Wanted Skills:
      • Mainframe COBOL / DB2 / CICS programming experience
      • Development experience using Smalltalk Seaside framework for web applications
      • Java
      • VA Smalltalk IDE
Good luck with your job hunting,
James T. Savidge

View James T. Savidge's profile on LinkedIn

This blog’s RSS Feed

Filed under: Employment Tagged: jobs, Smalltalk, Smalltalk jobs

Benoit St-Jean - Les casse-tête

Ceux qui me connaissent bien le savent, j’adore me casser la tête sur une foule de « petits » problèmes (mathématiques, algorithmiques ou autres) : ça permet de garder le cerveau en forme et ça me donne une occasion de faire du Smalltalk et de me garder à jour dans mes skills autant de programmation que d’analyse.

Si vous êtes comme moi, voici une liste de ces petits casse-tête qui m’amusent en ce moment (ou depuis un bout) et qui pourrait vous servir de suggestions…

Les nombres de Lychrel

Avant tout, un peu de vocabulaire!

Un palindrome est une mot, une phrase ou un nombre qui s’écrit de la même façon à l’endroit et à l’envers.  Par exemple, Laval, Bob ou 17371.  Ça peut également être une phrase ou un bout de texte comme « Mon nom » ou le célèbre « A man, a plan, a canal: Panama ».

Grosso modo, un nombre de Lychrel est un nombre qui ne peut pas former de nombre palindrome lorsqu’on l’additionne à répétition avec son « inverse ».

Par exemple, 59 n’est pas un nombre de Lychrel puis qu’on aboutit à un palindrome au bout des itérations suivantes:

59 + 95 = 154
154 + 451 = 605
605 + 506 = 1111

L’histoire devient passionnante quand des mathématiciens se sont intéressés au plus petit nombre pour lequel on n’a pas encore trouvé de nombre palindrome après une quantité anormalement élevée d’itérations.  Il s’agit de 196.

Le problème est fascinant en soi de par la simplicité des opérations impliquées (des additions et la détection de palindrome) et les raccourcis et astuces nécessaires (autant mathématiques qu’algorithmiques) pour tenter de le résoudre.

Parmi les incontournables sur le sujet, il y a les sites de Jason Doucet et celui de Wade VanLandingham.

Le site What If?

Un site de questions absurdes avec des explications sérieuses et scientifiques.  Un challenge pour le cerveau quand on essaie de formuler une réponse aux problèmes présentés! Ce qui est intéressant, c’est le raisonnement et les arguments des réponses à des problèmes aussi hypothétiques que, bien souvent, idiots!  Comme celui-ci par exemple.

La persistence multiplicative

On définit grossièrement la persistence multiplicative par le nombre de fois qu’on peut multiplier les chiffres d’un nombre entre eux jusqu’à ce que le résultat ne comporte qu’un seul chiffre.

Par exemple:

679 ->  6 * 7 * 9 = 378
378 ->  3 * 7 * 8 = 168
168 ->  1 * 6 * 8 = 48
48 ->  4 * 8 = 32
32 ->  3 * 2 = 6

On dira donc que le nombre 679 a une persistence multiplicative de 5.

À ce jour, nous ne connaissons aucun nombre en deça de 10^233 (10 à la puissance 233) ayant une persistence multiplicative supérieure à 11. C’est précisément cette limite qui intéresse les mathématiciens!

Évidemment, ce qu’il est intéressant de chercher ce sont les nombres dits candidats.  Tout nombre comportant un 0 est éliminé d’office (ça occasionne forcément un résultat de zéro). Comme la multiplication par 1 n’apporte rien de plus, on ne considérera que les nombres sans le chiffre 1.  De plus, on éliminera les nombres comportant à la fois le chiffre 5 et un nombre pair comme cela produira un multiple de 10, donc un résultat de zéro.

Pour un bref aperçu de la persistence multiplicative, il y a une page de Wolfram sur le sujet. Pour un résumé des optimisations et trucs possibles, il y a ce papier.

La conjecture de Collatz

Un autre problème mathématique non résolu : la conjecture de Collatz (aussi appelée conjecture de Syracuse).

Si vous avez quelques cycles de CPU à partager, il existe un projet BOINC juste ici.

Autrement, je vous recommende de lire sur le sujet.  Les diverses astuces pour accélérer les calculs sont aussi surprenantes qu’efficaces!

Euler Project

Un site qui propose des problèmes mathématiques à être résolus par ordinateur!  Il y a un hic!  Il devient assez rapidement évident que la solution est bien souvent (pour ne pas écrire toujours) algébrique et mathématique.

C’est ce dont je me suis encore rendu compte récemment, en scrappant toute une nuit à faire des calculs pour résoudre le problème 131!


Il y a tant d’articles sur le sujet qu’un simple recherche Google devrait suffire à vous tenir occupé en lecture jusqu’à la fin des temps!

Pour ma part, par pur amusement, je me suis attardé à résoudre ces petits problèmes d’une façon surprenante : le préambule, le premier article, la seconde partie et la dernière partie.

Pour un aperçu des techniques de résolution (autres que la force brute), il y a cette liste.

EinStein würfelt nicht!

Communément appelé EWN, ce petit jeu renforme une quantité de particularités contre-intuitives. Autre difficulté, l’aspect probabiliste qui vient tout brouiller les cartes!

Vous pouvez y jouer sur le site de jeux Little Golem si ça vous intéresse.  Pour ma part, je planche sur quelques idées de stratégies que j’entends bien tester avec un programme bientôt.


Un petit jeu tout simple!  Et pourtant!

J’ai commencé à étudier de plus près le ruzzle comme en témoigne cet article et celui-ci.

Comme je n’ai toujours pas trouvé de grille meilleure que celle de M. Müller, ma quête se poursuit! Des nouvelles pour très bientôt!


Classé dans:algorithmes, bases de données, informatique, jeux, mathématiques, MySQL, optimisation, Pharo, programmation, Smalltalk, sudoku Tagged: 196, algorithmique, analyse, BOINC, conjecture de Collatz, conjecture de Syracuse, Didier Müller, EinStein würfelt nicht!, Euler Project, EWN, Jason Doucet, Little Golem, Lychrel, mathématiques, optimisation, ordinateur, palindrome, persistence multiplicative, problèmes, programmation, projet, Ruzzle, Smalltalk, Sudoku, Wade VanLandingham, What If, Wolfram

Clément Béra - Introducing Immutability in the Cog VM

Hi everyone,

I have not written on my blog for a long time and I am sorry about that. There is a good reason though. I have been working for a while on a book which teach to Smalltalk developers how to implement a simple object virtual machine. One part of the book, dealing with call stack management, has moved to a closed beta and a few testers are trying to do the tutorials. Hopefully this part of the book will be available in the mid-january to everyone (as open beta).

During the last month, I tried to work from time to time on introducing Immutability in the Cog VM. The Cog VM was finally compiled with full immutability support last wednesday. I did most of the work myself, though as always, Eliot Miranda was very helpful. This post discusses the implementation of immutability in the Cog VM.

Immutability design

As argued on the virtual machine mailing list, we are talking about a write-barrier more than immutability itself.

My work was directly inspired from the immutability experiment implemented on the Newspeak interpreter which was used a few years ago (now the Newspeak interpreter and the Smalltalk interpreter are merged in the Cog VM) and is somehow similar to VisualWork’s immutability.

Two main new primitives are introduced. One  (isImmutable) allows to check if an object is immutable. The other one (setImmutabilityTo:) makes an object mutable or immutable based on the boolean argument.

Once an object is set as immutable, any primitive attempting to change the object’s state and instance variable stores fail.

Failing a primitive in case of immutability is easy to implement as the Cog VM already support primitive failures. The in-image primitive fall-back code may require to be edited to check if the primitive failed because of immutability and have a different behavior if so (for example raising a NoModification error). There is nothing much to say about that.

VM Callback

The main issue lied with instance variable store failure. The generic idea is to trigger a virtual machine callback, similarly to #doesNotUnderstand: , when an instance variable store is performed on an immutable object. The call-back notifies the programmer that the program is attempting to mutate an immutable object before the mutation has happen. Then, the program can do different things, such as raising an error or change the object to mutable state to perform the store.

VM callback are already supported, but none of them are similar to the new one we introduced for instance variable store failure, #attemptToAssign:withIndex: . This VM callback happens in an instance variable store instruction.

One problem is that the context’s pc will be after the instance variable store when the call back happens, with a given stack state, and that given stack state does not expect anything to be pushed on stack before the next instruction. The call back #attemptToAssign:withIndex:, as any message send, should return a value that would be pushed on stack, but if that happens, the sender’s stack would be messed up.

To solve this problem, I designed the #attemptToAssign:withIndex:: callback as a method returning no value. This can be done hacking the active process. Here is an example code of #cannotAssign:withIndex: :

attemptToAssign: value withIndex: index 
"Called by the VM when assigning an instance variable of an immutable 
Upon return, executing will resume after the inst var assignment. If 
the inst var had to be performed, do it manually here in the call back 
with instVarAt:put: . 
This method has to return *no* value. I do it by hacking the process 
(as for sista callbacks) until we provide a better solution." 
| process | 
"self do something here if you want." 
process := Processor activeProcess. 
[ process suspendedContext: process suspendedContext sender ] 
    forkAt: Processor activePriority + 1. 
Processor yield. 


As you can see, the method returns no value as the sender does not expect any value to be returned. The sender could, for example, have an empty stack and expect the next instruction to be performed on an empty stack.

Another problem lies with the machine code version of the method. Because of the call-back, the execution can be interrupted on any instance variable store. I will discuss later in the article the implications.

Immutability exception

I wanted the immutability design to be simple to hit production as fast as possible. For this purpose, I needed to select objects that can’t be immutable because they require a lot of additional work. In the future, I could change the VM to allow these objects to be immutable if someone can show me a production application that requires it. It is just harder to get these objects immutable, so I didn’t do it.

I distinguish two kind of objects that can’t be immutable for now:

  • Context: Contexts represent method and closure activation records. There are difficult to make immutable as they are mapped to stack frames in the VM for performance.
  • Objects related to Process scheduling: The virtual machine switches from a Process to another one from time to time while executing Smalltalk code. These switches implies that the Processor (global object) has to be mutated in the virtual machine. It can be tricky to execute an in-image callback when such objects are mutated as the VM is in the middle of a Process switch. For this purpose, I forbid Semaphores, the Processor, the linked list of Processes and the Processes themselves to be immutable. This decision was quite aggressive, but as I said before, I could consider letting more objects to be immutable if one shows me a good production application use-case.

Immutability in the memory manager

The first thing to do was to change the Memory Representation so each object keeps a specific bit to mark if they are immutable or not.

Fortunately, the Spur Memory Manager was designed with immutability in mind. In each object’s header, a bit was already reserved for immutability. Most of the code related to this bit (is the object immutable, set the object mutability) was already implemented, and I’ve just had to extend and bug-test it. Once this was ready, I had just to integrate the use of the immutability bit checks in the rest of the VM.

I didn’t implement immutability in the SqueakV3 Memory Manager. Immutability requires a Spur VM. We have not dropped support for this old Memory Manager, but there won’t be any development there anymore. Everyone should migrate to the Spur Memory Manager at some point.

Immutability in the interpreter

As stated before, this work was directly inspired from the Newspeak interpreter experiment. I used the code of the experiment as a reference for my work.

Two main things were done:

  • Instance variable stores: bytecode execution of instance variable stores were changed to check if the object mutated is immutable. If the object is indeed immutable, then the new callback I introduced in the interpreter (#attemptToAssign:withIndex:) is called.
  • Primitives: I went carefully through each primitive and fail them if they mutate an immutable object, at the exception of objects that can’t be immutable. In this case, a regular primitive failure happens though the error code may be different from other failures. I might have forgotten a few primitives, we’ll see if I get some bug reports.


Immutability in the JIT

Here we come to the real deal. How to make the JIT immutability compliant ?

As for the interpreter, I started with the abstraction over the memory representation of objects. I added the generation of machine instruction checking if an object was immutable or not. This was fairly easy as one just needs to check a bit in the object’s header.

Then I needed to change the primitives generated to machine code by the JIT to fail if they attempt to mutate an immutable object. There were only two primitives that needed to be changed: #at:put: and #stringAt:put: . Other primitives mutating objects are not compiled to machine code by the JIT. The change consisted in checking if the object mutated was immutable at the beginning of the primitive, in which case, the execution fallbacks to the C version of the primitive.

The second thing to do was to change the instance variable stores to check if the receiver was immutable. This was quite tricky.

The first problem there was that instance variable stores did not flush the register state. Registers used to be able to be live across an instance variable store. This does not work any more as an instance variable store can now trigger an in-image callback (basically a message send) which requires the registers to be flushed.

Unfortunately, there are no good solutions around it. Either the JIT flushes all the register and they can’t be live across the store, generating lots of spills on the main execution path, or the JIT restores the register state after the callback, which implies the generation of many more machine instructions per instance variable store, which can be even slower than flushing the register state.

I decided to flush every register at each instance variable store but the one holding the receiver. Hence, only the receiver register needs to be restored after the callback (one extra machine instruction). This allows the most common register, the one holding the receiver, to be live across instance variable stores.

The second problem lied with the callback. I needed to add a trampoline from machine code to the C runtime to trigger the #attemptToAssign:withIndex: callback. By convention, trampolines in the Cog always pass all their arguments by register. Instance variable stores used to put the value to store in any register available. That was not possible any more, as the trampoline needs a fixed register. I changed all the code related to the stores to fix the object mutated in a register (%EDX in x86) and the argument in another one (%ECX in x86). Then I added the trampoline, expecting the values in specific registers.

Lastly, each instance variable store instructions now needs to be mapped, i.e., the JIT needs to remember how to map the machine code program counter (mcpc) to their bytecode program counter (bcpc). This is important as, for example, the debugger can be opened in the #attemptToAssign:withIndex: callback requiring the context failing to perform the instance variable store to be displayed correctly. It can also be required for other operations, such as the divorce of all the stack page’s frames if the stack page needs to be freed.

This last point was easy to do in the NewspeakV4 and the SistaV1 bytecode set. However (and unfortunately), the default bytecode set of Squeak and Pharo is still SqueakV3PlusClosures. This bytecode set is always problematic. The main problem I had was that all the extended stores (stores over 16 encoding such as pushTemp: 20) are compiled in a single extendedStore instruction or with the DoubleExentedDoAnythingBytecode. This implies that if I map the bcpc to mcpc of the extended store instruction, all the store temp would be mapped too. Not only it implies mapping lots of instructions, but it also implies the mapping of the extendedStore instruction used specifically in case of primitive error code and compiled differently in the JIT… And I am not even talking about the DoubleExtendedDoAnythingBytecode, which is the craziest bytecode of this set.

Well, this was messy, but Eliot and I built some work around and it’s up and running by now. I hope soon some clients as Pharo will use by default the SistaV1 bytecode set, it will solve many issues (such as objects with over 256 instance variables, long jumps and many more) and simplify the VM work.


Well, now that immutability is up and running, I hope some of the Cog clients will use it. To try it, compile a VM with IMMUTABILITY set to true (it’s a C Compiler setting) and try to load the package here in one of the Smalltalk client. For the newspeak support, I guess one has to port it form the Smalltalk version. I have also started to discuss how to integrate Immutability in Pharo here. If someone is interested in making compiled method literals immutable, I will be happy to help.


January 23, 2016

Pharo Weekly - How to parse ndjson in Pharo with NeoJSON

Reading the ‘format’ is easy, just keep on doing #next for each JSON expression (whitespace is ignored).
| data reader |
data := ‘{“smalltalk”: “cool”}
{“pharo”: “cooler”}’.
reader := NeoJSONReader on: data readStream.
Array streamContents: [ :out |
  [ reader atEnd ] whileFalse: [ out nextPut: reader next ] ].

Preventing intermediary data structures is easy too, use streaming.
| client reader data networkStream |
(client := ZnClient new)
  streaming: true;
  url: ‘’;
networkStream := ZnCharacterReadStream on: client contents.
reader := NeoJSONReader on: networkStream.
data := Array streamContents: [ :out |
  [ reader atEnd ] whileFalse: [ out nextPut: reader next ] ].
client close.

It took a couple of seconds, it is 80MB+ over the network for 50K items after all.