Rafał,

Thanks for the update. We are glad you find Bamboo's Maven 2 support useful.

Cheers,
Anton

Here's link to my project's Bamboo installation, if anyone wants to see it in action: http://objectledge.org/bamboo/

I have installed Bamboo 2.6.2 and set up CI for my open source project consisting of about 30 modules with several layers/clusters of dependencies. I had a very good overall experience. The dependencies were extracted automatically, and build queue behavior with respect to dependency hierarchy was correct as far as I can tell at this point.
The only problem I had was that parent projects and maven2 plugins provided by my project were not recognized as dependencies, but I was able to work around this. See BAM-6906.
In my opinion maven2 support in Bamboo is now at least on par with Hudson.

feedback? been on Hudson for months now. It's fantastic!

2010/1/21 Mark Chaimungkalanont [Atlassian] (JIRA) <jira@atlassian.com>

Folks,

2.5 is now available and has this feature in it.

Please try it out and give us feedback, good, bad and ugly so we can continually improve it.

Cheers,

Mark C

Yes I had considered using the embedder, I'd used the Xpp3Reader to allow parsing as a simplification step as I needed something working fast.
I was originally storing the artifact coordinates separately with a view for exposing them in the project summary and/or allowing searching by each element, but never had the time.

Richard,

thanks for your input! It's great to hear that it's possible to handle at least some of the requirements with a plugin. This can serve both as a stop-gap measure until Atlassian decides to take care of this properly and serve as a development / testing branch for the core functionality that will be needed in the complete solution.

Two remarks: "GAV" is called "artifact coordinates" in Maven-speak, and it can actually include more elements that GroupId, ArtifactId, Version. See my comment #131581 (http://tinyurl.com/cuyzbc) above. Also, I think you don't need to store separate custom property for each dimension of the coordinates. Single structured string would do just fine.

Second, have you considered using embedded Maven to process your POMs? On one hand it can be an overkill, on the other, you can shield yourself from a lot of complexity. You cannot go wrong using the real thing to do your parsing.

I have implemented a plug-in for our internal system that achieves most of the requirements. I need to seek approval from my management before contributing the code, but I can outline the process at this stage.

This is split into two processors, the SnapshotDependencyProcessor (implements CustomBuildProcessor) runs on the remote agent (i.e. the box that actually executed the build), and the SnapshotDependencyPostProcessor (implements CustomBuildProcessorServer) runs on the build manager (the central manager that apportions builds to the agents).
The process is split because the build agent doesn't have access to persist any data between builds, it can just send back a Map of metadata, whereas the manager doesn't have access to the POM (which was checked out from the SCM on the build manager)

1) The SnapshotDependencyProcessor that runs on the remote agent first uses the MavenXpp3Reader to read the POM, then extracts the groupId, artifactId, and version for the parent and all dependencies (at present the process doesn't handle dependencies defined within a profile, but that would be a case of beefing up the Maven POM parsing as mentioned above). I assume that if there are any snapshots in the parent, the parent will also be a snapshot, so the build can rely on the parent's build being triggered (transitive builds if you will).
The SnapshotDependencyProcessor persists data in the build plan to identify the current project's groupId, artifactId and version (the identifiers for a project in Maven, called GAV from now on). It also processes the project's dependencies and parent and persists the GAVs for any SNAPSHOTs found.

2) The SnapshotDependencyPostProcessor compares the GAVs of all snapshots for the latest build with the GAVs for the previous build and if there are any differences (additions/deletions), locates the corresponding build plans (if any) and registers/unregisters dependencies with the corresponding plan as required.

Identifying a corresponding build is managed by checking the GAV values persisted in the custom properties by the SnapshotDependencyProcessor

for (Iterator iter = allBuilds.iterator(); iter.hasNext() {
Build projectBuild = (Build) iter.next();
Map customConfig =
projectBuild.getBuildDefinition().getCustomConfiguration();

if (groupId.equals(customConfig.get("custom.maven.groupId"))
&& artifactId.equals(
customConfig.get("custom.maven.artifactId"))
&& version.equals(customConfig.get("custom.maven.version")))

}
Registering the child build is handled by
dependentBuild.getChildBuilds().add(build);
getBuildManager().saveBuild(dependentBuild);

While the removal is handled by:
dependentBuild.getChildBuilds().remove(build);
getBuildManager().saveBuild(dependentBuild);

The custom properties are all copied from the CurrentBuildResult's custom data into the build definition's custom configuration to be used as the basis for the comparison for the next build (and to identify the plan as a dependency).

This approach means that, once each project has been built once, custom properties have been persisted so it can be identified as a dependency. Then subsequent builds will establish the full dependency tree over the next few builds.

Thanks for your comments Edwin. It certainly helps to reduce frustration levels to have considered comment on the issues.

Of course I appreciate that Atlassian has its own priorities which I am not privvy to, but I do still find some of the past prioritisations rather odd - adding bells and whistles rather than addressing core functionality.

For example:

• jira integration is nice (BAM-2974, BAM-2973, BAM-2971) , but surely the abiltiy to bulk edit plans (BAM-1182) is something that is worth more to a build tool
• and while "Elastic Bamboo" (BAM-3335) certainly sounds nifty, I can't imagine it would be of any use to your corporate customers, whom I assume, are paying your bills. maven integration, however, seems to me fundamental.

cheers,
Craig

Hi guys,

I completely understand your frustration around this, and agree that this is a feature that we do want to address in Bamboo. However, please do understand that Bamboo as a product has many priorities. Whilst the voting mechanism here has a strong influence on which features we tackle first, it is not the only source that we use to prioritize our features.

With regards to scheduling, we typically do not schedule hard fix for revisions against any version except the one that we are working on. This is the reason why no concrete fix for versions against this issue has been assigned yet. Unfortunately, we have taken much longer than we anticipated with the current 2.2 release. However, we are not too far away from completing that now. No absolute guarantees, but I highly anticipate that this will be available in the release after Bamboo 2.2.

We really do appreciate your patience on this issue, and thank you for giving us this kind of feedback to keep us honest.

Cheers,
Edwin

Atlassian, what are you doing to your customers? This is not the legendary service you promise (and you're achieving with other product). And definitely not Scrum. Why are you scheduling so many features (27) for upcoming 2.2 and do not include THIS? You should drop all features and only work on this one.
It becomes hard for me to present the advantage of Bamboo over other ci-server to my team and my customer with this feature missing.

Just to let you know that my client switched focus on Hudson now. I stop watching this issue and I also unvoted. Hudson will be it!

This is #1 voted issue in Bamboo, and yet it is still unscheduled. Come on guys, this is not rocket science and it will boost usability of Bamboo considerably.

Yep - it works smoothly with Hudson. Too bad that's still not available for Bamboo. Folks, vote for it!

I agree with Steve. A phased approach that tackled local Maven 2 builds first would give us exactly what we need.

Note: I only came looking for this feature request after discovering that Hudson already implements it: http://weblogs.java.net/blog/johnsmart/archive/2008/11/managing_automa.html

I second the motion to implement this functionality in phases, first for local builds only, and for maven 2 only. That would be of great use and hopefully this limitation of scope would allow it to be completed more quickly.

In my opinion the API between the Builder (doesn't matter if it's local or remote) and Bamboo core should be extended with a function for querying the "artifact designator" for the module being built, and artifact designators for all modules it depends on. It seems simple - one way communication passing a string + a list of strings. This information should be re-queried each time the module's sources are updated (on SCM trigger request or otherwise), resource designator <-> plan key mapping for the module updated, and dependencies for ALL plans hosted in Bamboo instance regenerated (if the administrator opted in for automatic dependency management of course). Changing the artifact designator of a module may cause a number dependencies of other modules on the update module be created or removed.

Full artifact designator syntax for Maven 2 is actually groupId:artifactId:packaging:classifier:version, but packaging and classifier elements are optional. Of course the syntax of designators should be completely opaque to Bamboo core, but that's something to keep in mind while implementing designator support in Maven 2 builder.

Sometimes a project needs to support Maven 1 and Maven 2 (and possibly other?) builds in parallel. There are 2 consequences of this 1: mapping needs to be kept at build plan rather than project level (I'm stating the obvious here) and 2: the artifact designator should include the builder's type identifier, for example as the first element akin to URL schema. This way Bamboo core can treat those as opaque strings and still successful Maven 2 builds will trigger only dependent Maven 2 builds and Maven 1 builds will trigger only Maven 1 builds.

I like the sound of that 2-phase suggestion.
The users of our Bamboo instance would also greatly benefit from having this enabled for local agents only.

Edwin,

My customer doesn't have remote agents, but the current build server is powerful enough to support multiple local builders. Can you implement that plugin in 2 phases? A first phase where this is supported in local build agents would already of tremendous value!! If you don't want to put this in a release without support for remote build agents, please get in touch with me to test out the Maven2 support for local build agents.

I will repeat: the current setup already has 202 Maven2 modules for one of their internal software systems. Another system is being migrated to the same build system as we speak and that is going to be another 200 to 300 modules! Currently, it is Continuum that builds these modules, but it does it only sequentially. Building in parallel would be a huge time saver!

Greetings,

Ringo

Hi Ringo,

Unfortunately, as Rafal points out, the current REST API in Bamboo doesn't support updating of dependencies, which will make it rather difficult to implement as a Maven plugin.

However, we have been thinking of the possibility of doing this as a Bamboo plugin which will work along these lines:
1) Bamboo will store a mapping of plan key to groupid:artifactId:version
2) Using xwork action (in the Administration section say), a function will read the pom.xml of maven 2 plans, and update the mapping. Using the mapping, this will also update the dependencies.
3) After each build/change on a maven 2 plan, we repeat (2).

The challenge however, is with remote builds. In this case, the local server doesn't have a copy of the pom.xml. We are looking at ways to overcome this.

Cheers,
Edwin

Ringo,

I checked the REST API documentation contained in my "test drive" Bamboo installation (version 1.2.4) and at this point it provides the following operations:

1. login
2. logout
3. updateAndBuild (SCM checkout & build)
4. executeBuild (manually trigger a build)
5. listBuildNames (discover projects & build plans)
6. listUsersBuildNames (discover user's favorite projects)
7. getLatestBuildResults (check status of discovered builds)

Frankly, this seems to be a sensible list as far as typical tool integration scenarios are concerned: SCM triggers, WWW status gadgets, taskbar gadgets, IDE plugins.

Rafal,

I expect from Bamboo the most primitive transactional behaviour: the build is done with the latest known build plan setup. When the build plan is changed while a build is taking place, the new setup is used for the next build. If I change the dependencies in a build plan using the web frontend, I would expect that this just works. I don't see why the Remote API or the web frontend difference should matter.

Nevertheless, can someone from Atlassian confirm that I can rely on such transactional behaviour?

Ringo

Edwin,

It seems that you must be developing the common modules with rock-solid API & behaviour consistency. Whenever a new build of common module is published that breaks test (or even compilation) in one of the dependent modules, this is not noticed until dependent module is rebuilt. At this point backtracking the change in the common module can be hard or impossible because of later overlapping changes or the build becoming official released artifact. It seems to me like forfeiting the advantage of doing Continuous Integration altogether. Perhaps there is something about the way you use Maven2 & Bamboo that I don't understand?

Ringo,

that would require two things: the ability to update build definition WHILE it is being processed (sounds problematic to me), and the ability to discover what maven modules are available in the bamboo installation to make dependencies on. You'd have to access the pom.xml in the working copy of the module's sources to discover it's groupId:artifactId:version coordinates unless the API allows querying such information from specialized builders. In short, I hope this issue will be handled inside Bamboo - it'll just be simpler and more robust that way.

Edwin,

In the public Atlassian Confluence, I found a reference to the Bamboo Remote API. However, it only mentions that the full API documentation is available on your running Bamboo installation. However, I do not have a Bamboo installation. Can you tell me if the Remote API supports updating the dependencies between modules in Bamboo?

My path of thought is to write a Maven2 plugin that updates the parent/child plan dependencies in Bamboo using the Remote API when the module is built. Maven2 has a full API available to work with the dependencies. So, during a module build, I would use the Remote API to discover the current build plan dependency settings, check what the Maven2 POM lists as dependencies and update if there are differences.

I can define the set of Maven2 goals that Bamboo needs to invoke for building, right?

Ringo

Craig,

At Atlassian, we use Bamboo for all our products and components. We also encourage all our plugin developers to use Bamboo.

Dealing with dependencies in Maven 2 has not proven to be as complicated an issue for us. We use Maven 2 internally here, and we do have a large library of common modules that our products depend on. However, the common modules and products are typically implemented and tested quite separately from each other. This means that using Maven 2's dependency management has been mostly sufficient.

Cheers,
Edwin
(Bamboo Product Manager)

Hi guys,

Firstly, let me apologize for the lack of an update of our position on this. Moving forward, we will be trying to address these questions sooner and with more clarity.

Unfortunately, we currently do not have this feature scheduled.

Since this was raised, we have thought long and hard about the many ways we can deliver this. The ideal implementation we hoped to deliver would involve some major changes to the way projects and plans are configured. This has been the reason why this feature has been pushed back.

However, we do understand that this is currently the highest voted for issue, and in light of this, we are looking for and evaluating alternative strategies to deliver this. Unfortunately at this stage, I cannot give a firmer schedule as to when this feature will be delivered yet.

Cheers,
Edwin Wong
(Bamboo Product Manager)

I wonder how Atlassian themselves deal with CI, since I'm pretty sure they use maven, and from their product portfolio I'm guessing they have many more artifacts, and much more complicated dependency hierarchies than we do at my company.

Do they use bamboo company wide? I'm guessing no.

cheers,
Craig.

Dear Atlassian,

On Oct 15, 2007, Mark Chaimungkalanont commented (see higher) that this issue would be resolved in Bamboo 2.1. Like Mark wrote it, it seemed a clear commitment to me. However, version 2.1 came and nothing changed regarding this issue.

As a freelance consultant, I am working at a large Belgian company. One of their internal software products is comprised of about 200 Maven2 modules. My current project within that company is to help bring over another software product into the same build system, which would lead to another 300+ modules into the continuous integration tool. Our current CI tool is Continuum, the only tool that "understands" Maven2 dependencies. However, the CI tool will eventually need to manage 500+ modules. Continuum doesn't support distributed builds which is the main drawback of that tool. However, without Bamboo "understanding" Maven2 dependencies, it is not an option to switch to!

The company is a JIRA user already, so Bamboo is on the top of their list of tools supporting distributed builds. All entries on that list do not support Maven2 dependencies by the way.

A clear question I want an answer on is the status of this issue? Is or isn't Atlassian working on a resolution for this issue? A clear "no" is better than the usual "maybe", since I would then get together with some other people that commented on this issue to see if we could get some sort of a plugin together that already gives some limited support. A clear "yes", with an exact committed version and ETA is offcourse much better!

Enlighten us!

Greetings,

Ringo

+1

Ping.

This is the number one "popular issue" in bamboo at the moment and I was wondering if any progress was being made? This is causing my developers grief at the moment who are making frequent commits to multiple maven artifact projects, and in return, frequently getting "broken build" emails which can lead them on a wild goose chase. This is due to the build being broken, not through any relation to their commit, but due to dependencies being built in the wrong order, and/or at the wrong time.

Is there any progress being made in the open source world on this? If Atlassian is working on it, is it something like a plugin that could be open-sourced so we can test and contribute?

thanks.

Another vote for adding the ability to parse ivyfiles for dependencies also. I will also cover all the bases by requesting the Ivy folks to enhance Ivy to export ivyfiles as maven poms (right now Ivy can read but not write poms).
Currently we use CruiseControl with a plugin called IvyCruise to order builds using dependencies in Ivy files. But that plugin is broken by recent versions of CruiseControl, yet development has stopped on the plugin and it is no longer supported. So we're looking to replace Cruise, and Bamboo would integrate well with our use of Fisheye and Jira.

The following jelly markup prints out the module / identity information for M1

<goal name="dependencies">
<echo>MODULE IDENTITY</echo>
<echo>${pom.groupId}:${pom.artifactId}:${pom.currentVersion}</echo>
<echo>MODULE DEPENDENCIES START</echo>
<j:forEach var="dep" items="${pom.dependencies}">
<echo>${dep.groupId}:${dep.artifactId}:${dep.version}</echo>
</j:forEach>
<echo>MODULE DEPENDENCIES END</echo>
</goal>

for M2, packaging and qualifier should be added to disambiguate the artifacts.

Of course in a typical setup only a few of the dependency ids are expected to be matched to existing Bamboo plans - only the ones that belong to the same multi-module project. Plan interdependencies need to be established among those. All the ids that are not resolved must be pointing to external, binary dependencies of the project and can be safely ignored.

This functionality is crucial for any non-trivial modular project.

I was thinking what would be the simplest possible interface between maven / maven2 and bamboo and I came up with the following:

let's implement plugins for maven 1 and maven 2 that print out to the standard output information about the current module identity followed by the identity of all modules that it depends on. Something like that:

org.myproject:module-a:1.0.0:jar
org.myproject:module-b:1.0.0:jar
org.apache.commons:commons-logging:2.0.0:jar

As you probably recognized it's concatenation of groupId artifactId version and packaging. Implementing such plugin should be very easy - I think I can whip up maven 1 version in a few minutes. Note that this mechanism could be generalized to other project builders as long as they have a notion of unique module identifiers and module dependencies.

Those strings could be treated as completely opaque by Bamboo. They're only useful to the extent that they can be mapped to Bamboo projects/plans, and further processing can proceed from there.

One thing to note is that it's probably not a good situation if more than one plans yield the same string as their identity marker. You should not have two plans that generate the same maven artifact - it's not only waste of resources, but potential source of problems due to race conditions / mutual repository clobbering.

Every time that a change to pom.xml or project.xml is detected, or a new project/plan is created or deleted, all plan dependency relationships between plans need to be regenerated.

I'd be happy to help along with this.

The point is, you have to trigger build the reverse order of the dependencies defined in the poms. Assume you change something in the leaf of a dependency path, all parents should be rebuild to see if they broke.

I am a little confused regarding this as all Maven artefact dependencies are resolved by the repository, so as long as you have your maven modules being monitored and built by a CI server (at any level, either per module as per Continuum or grosser arbitrary groupings) there is no need to string builds together into dependency chains as the latest version of the required deps are in the repository (local or remote). Guess im missing something...

hi mark,

I think the BuildProcessorModule isn't the right place for triggering a build, since it is executed during the build-process. Or did I missed something?

this is exactly the same, as it is implemented in the cruisecontrol maven2dependencies. The Implementation uses the MavenEmbedder-API to parse the POM.

Using it as Build Trigger in bamboo should be quite simple.

Rather than reinventing the wheel, we can use Maven's own infrastructure to parse POMs and resolve dependencies.

Folks,

Just to provide some feedback on this issue.

At the moment we're currently bunkered down implementing Bamboo 2.0 which will include our most popular issue (BAM-293). After that is done, this will be the most popular issue and one that we'll attack for Bamboo 2.1.

Our current implementation thought are:

• Bamboo to automatically read the {{pom.xml}}s of a plan when you set it up for a list of candidate groupId / artifactId for that plan (A list since the plan can have multiple modules, e.g. the Bamboo plan has atlassian-bamboo-api, atlassian-bamboo-core etc).
• Bamboo also finds all Maven dependencies and tries to match up the groupIds & artifactIds to other Bamboo plans (effectively setting parent plans)
• When a build completes, the derived Maven information is used for triggering off dependent builds
• The Maven metadata is recalculated with every build (since you might've updated the pom.xml)

This could be done as a plugin. You'd have the write a Build Processor Module to parse the pom.xml, read and store dependency data. Then you'd need to write a Build Completed Module to actually use this information to fire off the depedant builds.

Since we're going to attack this after 2.0, and that we're going to be doing some work on the plugin framework. I wouldn't recommend putting in a huge effort to write your own plugin, unless you badly need it.

Cheers,

Mark C

hi,

currently we are using cruisecontrol for CI, but now I would migrate to bamboo. But actually this issue blocks the migration.

In the default CruiseControl-package there is a plugin maven2dependency. The only thing that the project admin must do, is to specify the pom.xml for the current project.

I need the same behaviour on bamboo. But because the pom.xml is almost on the same point in the project, the configuration for the maven2dependency-Trigger is very simple.

If you don't have time to implement this, I can do it. Is it possible as plugin?

regards

It is nice to see that this issue is getting popular.

In this case (extending the request to other build systems) I think it would be appropriate to create a mini plugin framework mainly consisting of a central dependency repository, the special callbacks for each project and for the succesfully build project. Sorry if I call it a framework, its only a little bit of API and a little bit of runtime behaviour...

We would need a "bamboo" object model for the dependency tree. Each project-plugin should inject its dependencies into the central repository when asked. Then a successful build could then fire a "build yourself" event upwards the dependency tree. Each project should be able to decide if building is needed upon event consumption, and maybe should be allowed to sink the event and stop the delivery to its dependant projects. I think this event should know the projects it has traversed to avoid endless builds.

The project plugin also must provide the translation between the different internal project unique ids and the bamboo id. The maven pom only knows about maven artifact and group ids, the plugin will not know the bamboo id of its dependencies. It would be a pain to have to store them in the poms. The object model should be able to store something like a native id for this purpose and provide means for queries.

Don't know if this should be a separate issue, but ideally this same mechanism could read ivy.xml files and associated dependencies.

Feedback from another customer:

Now that granular security has been added to Bamboo, this is the single biggest shortcoming in Bamboo. I'm considering writing a Bamboo plugin to address this issue, but don't have the time in the near future. Maintaining the same dependency info in POMs and plan dependencies is a serious pain.

Maven2 stores the dependent pom artifacts in the jars.

It comes to a problem if there are dependencies defined in the parent pom, since the parent pom is added to the classpath. So you must know where the repository is and then load the whole tree up to the root parent pom to be complete.

Another way would be to write a mojo (maven2 plugin) that outputs the artifacts into some xml file.

It could work this way:

• Bamboo runs mvn bamboo:exportFullDependency
• Bamboo creates/ updates a maven dependency registry with the generated target/whateverTheNameOfTheExportedDepsForBamboo.xml
• Bamboo defines a trigger that looks into this central registry and fires all projects that have this one as dep in it registry
• maybe reorder the builds in the queue if it makes sense (like pushing projects up the queue to avoid endless retrigering)

A maven2 project definition should have a "automatic trigger deps" checkbox in the configuration. so we could define build plans that trigger full builds and some that dont.

Would be an awesome feature anyway.

Feedback from a customer:

The big thing we're missing is Bamboo automatically picking up build dependencies using the Maven 2 poms (both for initial import of the project, and if this changes over time).

We have a bunch of builds that depend on each other in ways that inevitably change over time, and it's a pain having to set up the dependencies to avoid builds giving false failure reports, when that information is already in the pom.xml. It's also a pain trying to remember to update it when things change.