An Update On Composer

This weekend we have been busy hacking on Composer in our office together with Nils Adermann and Volker Dusch. We wanted to push the project forward a bit faster than the odd free evenings usually allow, and I would now like to introduce the changes we made.

Development versions handling

The former master-dev and similar *-dev versions we used to have were causing quite a few issues, so we decided to overhaul that behavior in a way that allowed us to get more consistency and fix a few long standing issues. For example dev versions can now be locked to exact commit revisions, and they will update to the latest revision when you do an update, no need to delete them from disk beforehand.

Basically dev releases are now simply branch names with a dev suffix – for numeric branches which are comparable – or a dev prefix for textual names that are not comparable, like feature branches and master. There is no way to specify the version manually anymore in your repository’s composer.json, since that was causing potentially dangerous issues with feature branches conflicting with the original ones.

If your package depended on a master-dev version, you should now depend on dev-master. If your package depended on something like the Symfony2 2.1.0-dev version, this one also is now dev-master since it is in the master branch. Older feature branches like 2.0-dev which is the 2.0 branch and not master are unaffected by this change.

This change will break many packages out there that rely on -dev packages of any kind, and we hope everyone will update their composer.json files as swiftly as possible to make the transition less painful.

The Packagist version database had to be reset for this change, so things will look at bit empty for a couple of hours while everything is re-crawled. None of the packages are lost and you should not have to do anything except having a bit of patience.

Dependency solver stability

Nils and Volker have been doing big progress on bugfixing and testing the solver. Those are mostly highly technical details that I will not dive into here. But long story short many old bugs should be fixed, and then some. It may obviously have introduced regressions, so if you encounter any issues please report them with your composer.json file so we can easily reproduce.

Documentation

Igor has spent quite a bit of time on documentation, which you can see on github for now, and which should be migrated to getcomposer.org soon.

Packagist / GitHub integration

Another great new feature coming from a pull request by Beau Simensen is the ability to let GitHub tell Packagist when you push new code to your repository. This should make package updates almost instant. It should be integrated into the GitHub Service Hooks soon enough, so if you don’t want to set it up by hand you can wait a bit, otherwise you can grab your API hook URL on your Packagist profile page, and add it in your repository.

Repositories configuration

It seemed that the way custom repositories are configured was confusing, so we took the chance to make it a bit clearer. Basically names are dropped and it’s all stored in a flatter structure that’s easier to remember. Documentation has been updated on Packagist.

All in all it has been quite a productive week-end and we will continue working on a few things today.

February 20, 2012 by Jordi Boggiano in Development, News // Tags: , 6 Comments

jsDay – A Family Reunion You Always Wanted To Attend

Last jsDay in Verona was a blast. Seriously. To Jordi and myself this was one of the most memorable conferences ever.

And I am not talking about the sessions. Of course the talks were good and the jsDay team managed to attract some of the finest speakers to talk about everything JavaScript. But so do other conferences. The wonderful thing about jsDay is the community you become part a of as soon as you enter Hotel San Marco. Speakers and attendees alike share this hotel for 3 nights and 2 days and you feel you are part of the family reunion you never dared dreaming of.

Like-minded people sharing their passion. Cocktails at the hotel pool. A welcome quiet time in the Hotel Spa. Late supper in some of Italy’s finest Restaurants. A good laugh at Powerpoint Karaoke. And an incredible sense of belonging to this family that greets you with open arms and open hearts.

At Nelmio, we are now looking forward to the next jsDay in Verona and we just felt obligated to tell you about this little gathering that feels a bit like ours and that you too could be part of.

I sure hope to see you there on May 16th-17th 2012.

January 31, 2012 by Pierre Spring in News // Tags: 2 Comments

Composer: Part 2 – Impact

In the first part of this post I introduced Composer & Packagist. If you are not familiar with them please read part 1 first.

Impact

In this second part I would like to talk about a few things Composer could do for you, and the PHP community at large, once it is broadly adopted.

Common APIs and Shared Interfaces

You may have noticed that quite a lot of people are talking of and asking for more interoperability and cooperation between frameworks. It seems some PHP developers finally got tired of reinventing the wheel. That is great news. One way to provide this interoperability is through shared interfaces. The two main candidates there in my opinion are logging and caching. Two boring things that should just work, and where you always need tons of flexibility and tons of different backends, drivers, or whatever you want to call those. Almost every major framework and CMS out there have their own implementations of that stuff, yet none of them support all the options since there are too many.

The PHP Standards Group, an open mailing list discussing these interoperability questions has seen a recent proposal for a Cache Interface. One question raised was: How can those interfaces be distributed in each project that uses or implements them?

This is where I see Composer helping. Composer supports advanced relationships between packages, so to solve this issue you would need three parts (read carefully):

  • The psr/cache-interface package contains the interfaces, and requires a psr/cache virtual package.
  • Implementors of the interfaces (many libraries) all require the psr/cache-interface and also provide the psr/cache virtual package.
  • A framework that needs a cache library requires psr/cache-interface and hints the interface in its method signatures.

Then the user of that framework comes in, decides that he wants to use the Doctrine\Common cache implementation for example. By requireing doctrine/common, the psr/cache requirement of the psr/cache-interface would be satisfied. Both doctrine and the framework would use the interfaces from the psr/cache-interface package. No code duplication all over the place and everyone is happier. All those require and provide have version constraints on them, so the interfaces can easily be versioned so that Composer will not let you install things that do not work together.

Plugin Installs for Frameworks and Applications

Composer is built to be embedded in other frameworks, CMSs or other applications. Some parts are still a bit rough for that use case, but it is something that will be supported and encouraged. Reinventing the package management wheel is another thing that really should stop. Who am I to say this you ask? It is true, we are building a shiny new wheel as well. Yet I take comfort in the fact that we are trying to build a generic solution which will work for everybody.

Packages are easy to build – for those who insist on not reading the first part of this post: you drop a simple composer.json file and add the VCS repository to packagist.org. The goal is that building packages should be accessible. I would love it if TYPO3, Drupal or WordPress to name a few would use Composer as a library internally to handle their dependencies. The list of required packages does not have to be in a composer.json file, it can sit in a database just fine. That would mean that suddenly the WordPress plugin you are developing could depend on an external library to do some work, and you don’t have to embed the whole library code in your plugin’s repository. Autoloading would make it work magically as long as everyone respects PSR-0. Which brings me to my next point.

Promoting Standards

A few months back I was on IRC and someone linked his new library, who or what it was does not matter. I just noticed he used a home-made autoloader and asked him why he was not following the PSR-0 standard. The answer was “I just use a smarter autoloader, with fallback feature“. Now that’s great, maybe his solution is smarter in the way that it allows files and classes to be anywhere. But it messes with everybody else. No one can use that library unless they declare another autoloader just for it. Autoloading should really be a commodity that you do not have to lose time fixing.

By adopting and promoting the standard, I hope Composer will help raise awareness about it. If you follow PSR-0, Composer autoloads your packages. If you don’t, you are on your own. The more users start to rely on this, the more they will get annoyed when a package requires manual configuration to be autoloaded, which will put some pressure on the PSR-0 offenders.

Promoting Code Re-use

It is probably obvious, but having easy to use package management means you will use it more, and the more it is used, the more people will re-use and share code. I really hope to see many libraries pop up out there instead of the massive frameworks we had until recently.

This shift is already happening, the larger frameworks like Symfony2 and Zend Framework 2 have decoupled their internal components and it is now possible to use pieces of them individually. They start to look more like the PEAR repository, which is an aggregate of libraries that work well together, some depending on each other, but not all.

Single libraries out there are great but I see some value in these larger organizations enforcing some quality guidelines on their own code-base. In a way they act like brands. You know that if you use one of their packages you can expect a certain quality.

Renewed Interest in PHP

Overall, I believe that libraries like Buzz, Imagine and others can create a sort of DSL on top of the (sometimes really bad) PHP APIs. Many people have criticized PHP as a language for its inconsistencies and awkwardnesses. Fine. I am not going to argue with that. But I hope many of those people, if they are being honest, will agree that PHP as a platform is great. It runs everywhere, it does not require much configuration, it has an immense developer base.

If we have enough libraries that abstract away some of the language issues, I strongly believe PHP as a platform will have a bright future.

December 20, 2011 by Jordi Boggiano in Development // Tags: , 19 Comments

Composer: Part 1 – What & Why

You may have heard about Composer and Packagist lately. In short, Composer is a new package manager for PHP libraries. Quite a few people have been complaining about the lack of information, or just seemed confused as to what it was, or why the hell we would do such a thing. This is my attempt at clarifying things.

This second part of this post, Impact, has now been published.

What is it?

The Composer ecosystem is made of two main parts, both are available on GitHub. The development effort is being led by Nils Adermann and myself (Jordi Boggiano), but we already have more than 20 contributors which I would like to thank a bunch for helping.

Composer

Composer is the command-line utility with which you install packages. Many features and concepts are inspired by npm and Bundler, so you may recognize things here and there if you are familiar with those tools. It contains a dependency solver to be able to recursively resolve inter-package dependencies, a set of downloaders, installers and other fancy things.

Ultimately as a user, all you have to do is drop a composer.json file in your project and run composer.phar install. This composer.json file defines your project dependencies, and optionally configures composer (more on that later). Here is a minimal example to require one library:

{
    "require": {
        "monolog/monolog": "1.0.0"
    }
}

If we look at the package publisher side, you can see that there is some more metadata you can add to your package. This is basically to allow Packagist to show more useful information. One thing that is great though is that if your library follows the PSR-0 standard for class and files naming, you can declare it here (see the last two lines below) and Composer will generate an autoloader for the user that can load all of his project dependencies.

{
    "name": "monolog/monolog",
    "description": "Logging for PHP 5.3",
    "keywords": ["log","logging"],
    "homepage": "http://github.com/Seldaek/monolog",
    "type": "library",
    "license": "MIT",
    "authors": [
        {
            "name": "Jordi Boggiano",
            "email": "j.boggiano@seld.be",
            "homepage": "http://seld.be"
        }
    ],
    "require": {
        "php": ">=5.3.0"
    },
    "autoload": {
        "psr-0": {"Monolog": "src/"}
    }
}

Composer is distributed as a phar file. While that usually works out, if you can’t even get it to print a help with php composer.phar, you can refer to the Silex docs on pitfalls of phar files for steps you can take to make sure your PHP is configured properly.

Packagist

Packagist is the default package repository. You can submit your packages to it, and it will build new packages automatically whenever you create a tag or update a branch in your VCS repository. At the moment this is the only supported way to publish packages to it, but eventually we will allow you to upload package archives directly, if you fancy boring manual labor. You may have noticed in the composer.json above that there was no version, and that is because Packagist takes care of it, it creates (and updates) a master-dev version for my GitHub repo’s master branch, and then creates new versions whenever I tag.

You can run your own copy of Packagist if you like, but it is built for a large amount of packages, so we will soon release a smaller tool to generate repositories that should be easier to setup for small scale repositories.

If you have no interest in using Packagist with Composer, or want to add additional repositories, it is of course possible.

Why(s)?

Why do I need a package manager?

There is a huge trend of reinventing the wheel over and over in the PHP world. The lack of a package manager means that every library author has an incentive not to use any other library, otherwise users end up in dependency hell when they want to install it. A package manager solves that since users do not have to care anymore about what your library depends on, all they need to know is they want to use your stuff. Please think about it real hard, let it sink, I will get back to that in the next post.

So we started working on Composer because there is no satisfactory solution at the moment for PHP, and that is quite unacceptable in this day and age. Of course with such a bold statement, you may be wondering:

Why not use PEAR?

While PEAR was and remains a viable option to some people, many have also been dissatisfied with it for various reasons. Composer has a very different philosophy, and it probably will not please everybody either. The main aspect that differs is that PEAR started as a system-wide package manager, much like apt-get or other similar solutions.

That approach does not work very well when you have many projects running on one machine, some of them 5 years old and depending on outdated versions of a library a newer project also uses. You can’t easily install both versions at the same time, and lots of frustration ensues.

Another issue is that one project’s dependencies become very fuzzy, since you code against code that is installed somewhere on your system, you can easily forget to mention in your README that your app depends on library X. Future-you or another guy comes along, tries to setup the project and is left in a run -> see error -> install lib -> run loop until all errors are gone. If he is really out of luck, he misses one dependency that is rarely used, and something fails unnoticed later on.

Composer on the other hand forces you to declare your project dependencies in a one-stop location (composer.json at the root). You just checkout the code, install dependencies, and they will sit in the project directory, not disturbing anything else on the machine. Another related feature is the composer.lock file that is generated when you install or update dependencies. It stores the exact version of every dependency that was used. If you commit it, anyone checking out the project will be able to install exactly the same versions as you did when you last updated that file, avoiding issues because of minor incompatibilities or regressions in different versions of a dependency. If you ever had bugs appear only on one team member’s machine while the others were fine because of some too-new or too-old version of something, you will know this is very useful.

Another notable difference, although some may not care about this, is that there is no approval process to have your package included on Packagist. While our vendor-name/package-name convention resembles PEAR’s channel/package, we do not have channels. All repositories contain packages that go into one big package pool, and then the solver figures out which packages fit your requirements, no matter where they come from.

Why JSON for packages?

It is a recurring question so I will answer it, hopefully for the last time. The short answer is because. The longer one is that there are many options (yaml, json, xml, php, ini, whatever.), each have their fan-base, and each have their haters. Whatever we would have picked, someone would be complaining. If you think it is a stupid decision, I am sorry to announce you are in the group selected to be the folks complaining, but it is not going to change. Please try not to focus on such a detail, and look at the bigger picture.

Where are we now?

I have delayed writing this post for quite a long time. I wanted it to be all nice and shiny before announcing anything. Unfortunately it is not as polished as I would like yet, but we are getting there. Composer can install itself (well, its dependencies) via Packagist, and many people have played with it and looking to integrate it in their work environments.

Here are the main points that still need some love, and of course if you would like to help you can join us on IRC (freenode #composer-dev) or on the mailing list.

Documentation

Documentation – or the lack thereof – is a huge problem right now. This post is a first step in that direction, and we will definitely work on more formal documentation in the future. You could too.

Solver bugs

The dependency solver is a complex beast, it has been ported from C code and it still has some rough edges. Not much to say here, we just need people to try it and report bugs. Bonus points if you can write a unit test that reproduces the issue.

Private repositories

This is a big topic that we need to address as well. Installing closed-source packages is of course necessary in most companies, and we will definitely work on it once the basics are working well and the open-source use case is covered. If you have some time or money to invest in that and want it to happen ASAP, please get in touch with us.

Global installs

As I said, we work with local installs by default, and that will not change for everything that is directly project-related. That being said, there is a whole set of CLI-tools for testing/QA or other purposes that would benefit from being installed system-wide and executable from anywhere. It is already possible to do a local install of those in your home dir and then add the bin directory of that install to your PATH of course, but we would like to support a more streamlined experience.

Part two will come next week, covering a few use cases, visions and hopes we have for Composer and PHP as a whole. To stay up to date you can follow @packagist or myself on twitter.

Update: You can now read Part2

December 8, 2011 by Jordi Boggiano in Development // Tags: , 47 Comments

CORS with Sencha Touch

A while back I was working on the mobile version of techup.ch which you can find on m.techup.ch. The web application is built with Sencha Touch. I did not want to deploy to m.techup.ch yet, which is where I ran into issues with the same origin policy.

In order to prevent web applications gaining access to things they should not have access to there is a same origin policy, which means that AJAX requests can only be made to the same host and port that the website is hosted on. This prevents CSRF attacks, as an attacker cannot make your browser do arbitrary actions on remote websites.

Our Sencha Touch app however relies on a JSON API, that is served from techup.ch. Since the app itself is not hosted on that domain (yet), it cannot access the API.

A workaround: JSONP

While AJAX is restricted by the same origin policy, script tags are not. It is possible to include script tags to a remote site, which is often used to embed widgets from sites like Facebook or Twitter. It can also be abused to fetch data from a remote domain.

This works by adding a script-tag proxy pointing to the API, including a callback parameter in the query string, for example:

1
http://techup.ch/api/events/upcoming.json?callback=processData

The API will detect the callback parameter and pad the response with a javascript call to the data it returns.

Conventional Reponse

{
    "events": [
        {
            "id": 361,
            "name": "Linalis LPI 201"
        }
    ]
}

JSONP Response

processData({
    "events": [
        {
            "id": 361,
            "name": "Linalis LPI 201"
        }
    ]
});

And this is what the “P” in JSONP is. JSON with padding.

Of course the processData function has to exist. In fact, jQuery’s AJAX function has built in support for JSONP. If you have callback=? in the requested URL, it will automatically use JSONP for the request. Sencha Touch also supports this through Ext.data.ScriptTagProxy.

But this is more a hack than a real solution. Enter CORS.

Cross-Origin Resource Sharing

CORS is a W3C Working Draft which describes an extension to HTTP for allowing browsers to issue AJAX request across domains. The server serving this request can send additional headers notifying the client of these additional permissions.

The only header that needs to be set is:

Access-Control-Allow-Origin: *

This will allow AJAX requests from any domain. Instead of the wildcard, you could also specify allowed domains explicitly. Only set the wildcard if it is safe to do so. You need to be aware that any site could get any user to make a request to those resources on your site that have this header, which can very easily lead to CSRF attacks.

Now, Sencha Touch (as well as many other AJAX libraries) will also send a X-Requested-With header, allowing the server to detect that the request was sent through JavaScript. CORS forces the server to specify which headers can be sent by the client. That is done as folllows:

Access-Control-Allow-Headers: x-requested-with

This is a comma-delimited list, so you can add more headers if needed.

If you want to allow other HTTP methods than GET, you’ll have to specify that explicitly with yet another header:

Access-Control-Request-Method: GET,POST

Authentication

By default the browser will not send any cookies with CORS requests. You can however set the Access-Control-Allow-Credentials header with a value of

1
true

, which will allow cookies to be sent. In this case you may however want to explicitly define a range of allowed origin domains.

Alternatively you can handle authentication yourself. In this case the user will have to provide username and password to the app, which will then make a request to the API. The API will respond with an authentication token, which can be used in subsequent authenticated requests to the API. This way you do not have to rely on cookies. You can store this token in localStorage, so you don’t loose it on page reloads.

Browser support

A recent article describes the browser support as follows:

  • Webkit browsers: good
  • Gecko browsers: good
  • Trident browsers (Internet Explorer 8+): good with some gotchas
  • Opera: very sucky a.k.a. non existent

Since we are targeting mobile here, which is in this case pretty much webkit only, we can just use it.

Conclusion

CORS is a nice solution for remote API access from mobile web applications.

Check out enable-cors.org.

November 3, 2011 by Igor Wiedler in Development // Tags: , , 13 Comments

Kickstarting a Swiss OpenData Revolution

Last weekend I took part in the MakeOpenData Camp. The goal of the event was on one side to encourage the government to open more data by showing that we can make good use of it. On the other side, events like this help to get more developers and designers involved and actually realize that there is some data that is publicly accessible. Continue reading

October 5, 2011 by Jordi Boggiano in News // Tags: , Comments Off on Kickstarting a Swiss OpenData Revolution

The Algorithm That Named Us Nelmio

I have never been one of the cool kids that play in a band, but I have heard that kids in a band spend more time coming up with a good name than actually playing music. My co-founder Jordi and I felt a bit like that in the first few months after deciding that we would start our own company. At that time we were still working a full-time job at Liip and had many side projects in addition to speaking at various conferences. We simply did not have enough energy to put into our new company. But whenever we met, we eagerly discussed new business ideas and fantasized about the hardware we would provide our employees. Sometimes we even worked on minor projects like Techup or Don’t Make Me Steal (did you sign the petition?). And there was one thing we always ended up doing, no matter what: Trying to find a good name for our company.

This is the story of how we came up with the perfect way to name a company! Continue reading

August 17, 2011 by Pierre Spring in News // Tags: 11 Comments

Introducing the NelmioSecurityBundle

At Nelmio we love Symfony2. As contributors to the core development, we care a lot about not only the project itself, but the entire ecosystem.

And that’s why we’re thrilled to announce the immediate availability of the NelmioSecurityBundle!

This Symfony2 bundle provides security-enhancing features for your application. It is not a replacement for the core SecurityBundle, it provides generic purpose security features, not related to user management. Continue reading

August 4, 2011 by Igor Wiedler in Development // Tags: , , 3 Comments

Symfony2 Released

Last week, Symfony2’s first stable release, 2.0.0, has been released. We are big fans of this modern PHP framework, and have been working with it and contributing to it since it was introduced at the Symfony Live 2010 conference a year ago. It’s great to finally see it come to life officially, and I can’t wait to see the feedback of the broader user base.

Now that it is finally stable, I would like to take the chance to announce that Nelmio offers in-house Symfony2 training for businesses. This way you can get a team up to speed quickly. We can also help you with quality assurance and code reviews, which is a great way to learn about the framework while looking at a real use case and helping your project move forward.

If you are an individual looking for training, please contact us. We will schedule a class as soon as there are enough people showing interest. Even better so, if you are around Zürich in September, you can take part in the one day workshop I am giving at the /ch/open which is quite affordable.

August 2, 2011 by Jordi Boggiano in Development // Tags: , Comments Off on Symfony2 Released

Igor Wiedler Joins the Team

We’re happy to welcome Igor Wiedler into our ranks starting in August!

He has been involved with phpBB development for a few years, but more importantly is core developer of the Silex micro-framework which is based on Symfony2 Components, which makes him a Symfony2 contributor as well.

You can find him on twitter and github.

July 19, 2011 by Jordi Boggiano in News // Tags: , 3 Comments