TAG composer

An Appeal to All Package Managers

I work on Composer – a new PHP dependency manager – and it is now working quite well for managing PHP packages. We have a decent amount (and growing fast) of packages on Packagist. All is well. Yet, most PHP projects are websites, and they need some frontend libraries, be it JavaScript or CSS – I will use jQuery as an example that everyone can grasp easily. In some frameworks, you have plugins that bundle a copy of jQuery. Some people have also used Composer to hack together jQuery packages so that they can download it, and then they have some scripts to copy or symlink the files in the proper location. That is all very flaky, you end up with multiple copies of jQuery and if you are lucky you even get various different versions.

I have been thinking about it for a few months, and it seems like nothing exists out there. Every language out there has its own package manager, but everyone seems to be stuck with the same problem of frontend code. Obviously jQuery is used by virtually everyone. They can’t support Composer, Bundler, npm, pip and god knows what. Building a package manager for JS/CSS could work, but the community is huge and scattered and getting broad adoption is going to be very difficult.

The plan

As far as I can see, and given the way Composer works, it would be fairly easy to build a central package repository – similar to Packagist – where people can submit package definitions for frontend libs. For instance I could go and add a twitter bootstrap package, define the URL where you can download it including some placeholder for the version. Then all we need is a list of version that’s maintained. We could start by doing that by hand, or we can just point it to the git repo and it reads tags. That’s how Packagist works – except that it reads the composer.json file in the git repo to get all the metadata that in this case would be entered manually.

If we do this, we then end up with a central repository of CSS and JS packages, and we can integrate it in Composer, so that Composer packages can depend on jQuery and it just works. That would be a good start, but the great thing would be to get everyone on board. And I don’t mean everyone writing PHP. I mean everyone. The Ruby folks, the Python ones, Java, .NET, you name it. You all have package managers. All we have to do is agree on the API of the central package repository and on what metadata is needed. Then you can just add support for it in your package manager of choice, and we all benefit from the manual work put in to list packages. If it works, I’m sure some of the frontend packages will then add the metadata directly in their git/svn/.. repos so that we save on manual work. This would be a huge thing for everyone.

There are of course a few more details to settle regarding security and trust as well as exact package metadata, but I wanted to gauge the interest first, and then discuss further. I opened a frontend-packaging google group for that purpose, so if you are interested please join in. All it takes is a few open minded people and we could start one of the largest cross-language collaboration project ever. Sounds like fun!

May 31, 2012 by Jordi Boggiano in Development // Tags: , , , , 8 Comments

An Update On Composer

This weekend we have been busy hacking on Composer in our office together with Nils Adermann and Volker Dusch. We wanted to push the project forward a bit faster than the odd free evenings usually allow, and I would now like to introduce the changes we made.

Development versions handling

The former master-dev and similar *-dev versions we used to have were causing quite a few issues, so we decided to overhaul that behavior in a way that allowed us to get more consistency and fix a few long standing issues. For example dev versions can now be locked to exact commit revisions, and they will update to the latest revision when you do an update, no need to delete them from disk beforehand.

Basically dev releases are now simply branch names with a dev suffix – for numeric branches which are comparable – or a dev prefix for textual names that are not comparable, like feature branches and master. There is no way to specify the version manually anymore in your repository’s composer.json, since that was causing potentially dangerous issues with feature branches conflicting with the original ones.

If your package depended on a master-dev version, you should now depend on dev-master. If your package depended on something like the Symfony2 2.1.0-dev version, this one also is now dev-master since it is in the master branch. Older feature branches like 2.0-dev which is the 2.0 branch and not master are unaffected by this change.

This change will break many packages out there that rely on -dev packages of any kind, and we hope everyone will update their composer.json files as swiftly as possible to make the transition less painful.

The Packagist version database had to be reset for this change, so things will look at bit empty for a couple of hours while everything is re-crawled. None of the packages are lost and you should not have to do anything except having a bit of patience.

Dependency solver stability

Nils and Volker have been doing big progress on bugfixing and testing the solver. Those are mostly highly technical details that I will not dive into here. But long story short many old bugs should be fixed, and then some. It may obviously have introduced regressions, so if you encounter any issues please report them with your composer.json file so we can easily reproduce.

Documentation

Igor has spent quite a bit of time on documentation, which you can see on github for now, and which should be migrated to getcomposer.org soon.

Packagist / GitHub integration

Another great new feature coming from a pull request by Beau Simensen is the ability to let GitHub tell Packagist when you push new code to your repository. This should make package updates almost instant. It should be integrated into the GitHub Service Hooks soon enough, so if you don’t want to set it up by hand you can wait a bit, otherwise you can grab your API hook URL on your Packagist profile page, and add it in your repository.

Repositories configuration

It seemed that the way custom repositories are configured was confusing, so we took the chance to make it a bit clearer. Basically names are dropped and it’s all stored in a flatter structure that’s easier to remember. Documentation has been updated on Packagist.

All in all it has been quite a productive week-end and we will continue working on a few things today.

February 20, 2012 by Jordi Boggiano in Development, News // Tags: , 6 Comments

Composer: Part 2 – Impact

In the first part of this post I introduced Composer & Packagist. If you are not familiar with them please read part 1 first.

Impact

In this second part I would like to talk about a few things Composer could do for you, and the PHP community at large, once it is broadly adopted.

Common APIs and Shared Interfaces

You may have noticed that quite a lot of people are talking of and asking for more interoperability and cooperation between frameworks. It seems some PHP developers finally got tired of reinventing the wheel. That is great news. One way to provide this interoperability is through shared interfaces. The two main candidates there in my opinion are logging and caching. Two boring things that should just work, and where you always need tons of flexibility and tons of different backends, drivers, or whatever you want to call those. Almost every major framework and CMS out there have their own implementations of that stuff, yet none of them support all the options since there are too many.

The PHP Standards Group, an open mailing list discussing these interoperability questions has seen a recent proposal for a Cache Interface. One question raised was: How can those interfaces be distributed in each project that uses or implements them?

This is where I see Composer helping. Composer supports advanced relationships between packages, so to solve this issue you would need three parts (read carefully):

  • The psr/cache-interface package contains the interfaces, and requires a psr/cache virtual package.
  • Implementors of the interfaces (many libraries) all require the psr/cache-interface and also provide the psr/cache virtual package.
  • A framework that needs a cache library requires psr/cache-interface and hints the interface in its method signatures.

Then the user of that framework comes in, decides that he wants to use the Doctrine\Common cache implementation for example. By requireing doctrine/common, the psr/cache requirement of the psr/cache-interface would be satisfied. Both doctrine and the framework would use the interfaces from the psr/cache-interface package. No code duplication all over the place and everyone is happier. All those require and provide have version constraints on them, so the interfaces can easily be versioned so that Composer will not let you install things that do not work together.

Plugin Installs for Frameworks and Applications

Composer is built to be embedded in other frameworks, CMSs or other applications. Some parts are still a bit rough for that use case, but it is something that will be supported and encouraged. Reinventing the package management wheel is another thing that really should stop. Who am I to say this you ask? It is true, we are building a shiny new wheel as well. Yet I take comfort in the fact that we are trying to build a generic solution which will work for everybody.

Packages are easy to build – for those who insist on not reading the first part of this post: you drop a simple composer.json file and add the VCS repository to packagist.org. The goal is that building packages should be accessible. I would love it if TYPO3, Drupal or WordPress to name a few would use Composer as a library internally to handle their dependencies. The list of required packages does not have to be in a composer.json file, it can sit in a database just fine. That would mean that suddenly the WordPress plugin you are developing could depend on an external library to do some work, and you don’t have to embed the whole library code in your plugin’s repository. Autoloading would make it work magically as long as everyone respects PSR-0. Which brings me to my next point.

Promoting Standards

A few months back I was on IRC and someone linked his new library, who or what it was does not matter. I just noticed he used a home-made autoloader and asked him why he was not following the PSR-0 standard. The answer was “I just use a smarter autoloader, with fallback feature“. Now that’s great, maybe his solution is smarter in the way that it allows files and classes to be anywhere. But it messes with everybody else. No one can use that library unless they declare another autoloader just for it. Autoloading should really be a commodity that you do not have to lose time fixing.

By adopting and promoting the standard, I hope Composer will help raise awareness about it. If you follow PSR-0, Composer autoloads your packages. If you don’t, you are on your own. The more users start to rely on this, the more they will get annoyed when a package requires manual configuration to be autoloaded, which will put some pressure on the PSR-0 offenders.

Promoting Code Re-use

It is probably obvious, but having easy to use package management means you will use it more, and the more it is used, the more people will re-use and share code. I really hope to see many libraries pop up out there instead of the massive frameworks we had until recently.

This shift is already happening, the larger frameworks like Symfony2 and Zend Framework 2 have decoupled their internal components and it is now possible to use pieces of them individually. They start to look more like the PEAR repository, which is an aggregate of libraries that work well together, some depending on each other, but not all.

Single libraries out there are great but I see some value in these larger organizations enforcing some quality guidelines on their own code-base. In a way they act like brands. You know that if you use one of their packages you can expect a certain quality.

Renewed Interest in PHP

Overall, I believe that libraries like Buzz, Imagine and others can create a sort of DSL on top of the (sometimes really bad) PHP APIs. Many people have criticized PHP as a language for its inconsistencies and awkwardnesses. Fine. I am not going to argue with that. But I hope many of those people, if they are being honest, will agree that PHP as a platform is great. It runs everywhere, it does not require much configuration, it has an immense developer base.

If we have enough libraries that abstract away some of the language issues, I strongly believe PHP as a platform will have a bright future.

December 20, 2011 by Jordi Boggiano in Development // Tags: , 19 Comments

Composer: Part 1 – What & Why

You may have heard about Composer and Packagist lately. In short, Composer is a new package manager for PHP libraries. Quite a few people have been complaining about the lack of information, or just seemed confused as to what it was, or why the hell we would do such a thing. This is my attempt at clarifying things.

This second part of this post, Impact, has now been published.

What is it?

The Composer ecosystem is made of two main parts, both are available on GitHub. The development effort is being led by Nils Adermann and myself (Jordi Boggiano), but we already have more than 20 contributors which I would like to thank a bunch for helping.

Composer

Composer is the command-line utility with which you install packages. Many features and concepts are inspired by npm and Bundler, so you may recognize things here and there if you are familiar with those tools. It contains a dependency solver to be able to recursively resolve inter-package dependencies, a set of downloaders, installers and other fancy things.

Ultimately as a user, all you have to do is drop a composer.json file in your project and run composer.phar install. This composer.json file defines your project dependencies, and optionally configures composer (more on that later). Here is a minimal example to require one library:

{
    "require": {
        "monolog/monolog": "1.0.0"
    }
}

If we look at the package publisher side, you can see that there is some more metadata you can add to your package. This is basically to allow Packagist to show more useful information. One thing that is great though is that if your library follows the PSR-0 standard for class and files naming, you can declare it here (see the last two lines below) and Composer will generate an autoloader for the user that can load all of his project dependencies.

{
    "name": "monolog/monolog",
    "description": "Logging for PHP 5.3",
    "keywords": ["log","logging"],
    "homepage": "http://github.com/Seldaek/monolog",
    "type": "library",
    "license": "MIT",
    "authors": [
        {
            "name": "Jordi Boggiano",
            "email": "j.boggiano@seld.be",
            "homepage": "http://seld.be"
        }
    ],
    "require": {
        "php": ">=5.3.0"
    },
    "autoload": {
        "psr-0": {"Monolog": "src/"}
    }
}

Composer is distributed as a phar file. While that usually works out, if you can’t even get it to print a help with php composer.phar, you can refer to the Silex docs on pitfalls of phar files for steps you can take to make sure your PHP is configured properly.

Packagist

Packagist is the default package repository. You can submit your packages to it, and it will build new packages automatically whenever you create a tag or update a branch in your VCS repository. At the moment this is the only supported way to publish packages to it, but eventually we will allow you to upload package archives directly, if you fancy boring manual labor. You may have noticed in the composer.json above that there was no version, and that is because Packagist takes care of it, it creates (and updates) a master-dev version for my GitHub repo’s master branch, and then creates new versions whenever I tag.

You can run your own copy of Packagist if you like, but it is built for a large amount of packages, so we will soon release a smaller tool to generate repositories that should be easier to setup for small scale repositories.

If you have no interest in using Packagist with Composer, or want to add additional repositories, it is of course possible.

Why(s)?

Why do I need a package manager?

There is a huge trend of reinventing the wheel over and over in the PHP world. The lack of a package manager means that every library author has an incentive not to use any other library, otherwise users end up in dependency hell when they want to install it. A package manager solves that since users do not have to care anymore about what your library depends on, all they need to know is they want to use your stuff. Please think about it real hard, let it sink, I will get back to that in the next post.

So we started working on Composer because there is no satisfactory solution at the moment for PHP, and that is quite unacceptable in this day and age. Of course with such a bold statement, you may be wondering:

Why not use PEAR?

While PEAR was and remains a viable option to some people, many have also been dissatisfied with it for various reasons. Composer has a very different philosophy, and it probably will not please everybody either. The main aspect that differs is that PEAR started as a system-wide package manager, much like apt-get or other similar solutions.

That approach does not work very well when you have many projects running on one machine, some of them 5 years old and depending on outdated versions of a library a newer project also uses. You can’t easily install both versions at the same time, and lots of frustration ensues.

Another issue is that one project’s dependencies become very fuzzy, since you code against code that is installed somewhere on your system, you can easily forget to mention in your README that your app depends on library X. Future-you or another guy comes along, tries to setup the project and is left in a run -> see error -> install lib -> run loop until all errors are gone. If he is really out of luck, he misses one dependency that is rarely used, and something fails unnoticed later on.

Composer on the other hand forces you to declare your project dependencies in a one-stop location (composer.json at the root). You just checkout the code, install dependencies, and they will sit in the project directory, not disturbing anything else on the machine. Another related feature is the composer.lock file that is generated when you install or update dependencies. It stores the exact version of every dependency that was used. If you commit it, anyone checking out the project will be able to install exactly the same versions as you did when you last updated that file, avoiding issues because of minor incompatibilities or regressions in different versions of a dependency. If you ever had bugs appear only on one team member’s machine while the others were fine because of some too-new or too-old version of something, you will know this is very useful.

Another notable difference, although some may not care about this, is that there is no approval process to have your package included on Packagist. While our vendor-name/package-name convention resembles PEAR’s channel/package, we do not have channels. All repositories contain packages that go into one big package pool, and then the solver figures out which packages fit your requirements, no matter where they come from.

Why JSON for packages?

It is a recurring question so I will answer it, hopefully for the last time. The short answer is because. The longer one is that there are many options (yaml, json, xml, php, ini, whatever.), each have their fan-base, and each have their haters. Whatever we would have picked, someone would be complaining. If you think it is a stupid decision, I am sorry to announce you are in the group selected to be the folks complaining, but it is not going to change. Please try not to focus on such a detail, and look at the bigger picture.

Where are we now?

I have delayed writing this post for quite a long time. I wanted it to be all nice and shiny before announcing anything. Unfortunately it is not as polished as I would like yet, but we are getting there. Composer can install itself (well, its dependencies) via Packagist, and many people have played with it and looking to integrate it in their work environments.

Here are the main points that still need some love, and of course if you would like to help you can join us on IRC (freenode #composer-dev) or on the mailing list.

Documentation

Documentation – or the lack thereof – is a huge problem right now. This post is a first step in that direction, and we will definitely work on more formal documentation in the future. You could too.

Solver bugs

The dependency solver is a complex beast, it has been ported from C code and it still has some rough edges. Not much to say here, we just need people to try it and report bugs. Bonus points if you can write a unit test that reproduces the issue.

Private repositories

This is a big topic that we need to address as well. Installing closed-source packages is of course necessary in most companies, and we will definitely work on it once the basics are working well and the open-source use case is covered. If you have some time or money to invest in that and want it to happen ASAP, please get in touch with us.

Global installs

As I said, we work with local installs by default, and that will not change for everything that is directly project-related. That being said, there is a whole set of CLI-tools for testing/QA or other purposes that would benefit from being installed system-wide and executable from anywhere. It is already possible to do a local install of those in your home dir and then add the bin directory of that install to your PATH of course, but we would like to support a more streamlined experience.

Part two will come next week, covering a few use cases, visions and hopes we have for Composer and PHP as a whole. To stay up to date you can follow @packagist or myself on twitter.

Update: You can now read Part2

December 8, 2011 by Jordi Boggiano in Development // Tags: , 47 Comments