1) Sessions aren't really a problem - most application servers support session clustering across their application servers. However, we need to ensure that:

• we don't put large objects into the session
• that all objects in the session are serializable

2) Moving attachments to the database is not a large problem.

3) Indexes are a large problem, as we use them for pretty much everything now (all statistics, graphs, searching). There would either need to be a central 'index server' which everything talked to (and it in turn talks to the database), or we'd need some event system to push changes out to individual indexes

4) The other part is application level caches - these need to be clustered. We currently have many application-level caches for things like projects, versions etc. We need to do 2 things here:

• Investigate where we can remove the need for these caches by using more efficient database access
• For the ones that can't be removed, we need to investigate some clustering mechanism for them, such as javagroups or tangosol.

If anyone has previous experience with this - please comment here.

• Why do you need an "indexing" architecture, why are database search and indexing capabilities not enough? (I ask this because in the projects I have seen all sorts of caching mechanisms have been developed and they all had bugs and performance problems and made the products more difficult to maintain and evolve)

• If you need these indizes anyway, why dont you put them in the database as LOB types? This way the database would do the synchronization and keep these indexes in a consistent state at any time. Moreover enterprise installations of JIRA would be able to use database clusters with more than one node and thus inherent failover capabilities (we are using ORACLE grids, real applications clusters and SQL server clusters).
  This would mean that only companies with such cluster databases could use this "enterprise feature" but that would be tolerable in my eyes.
  I think developers tend to "re-invent the wheel" especially in situations where a problem "is great fun / challenge".
  I was envolved in the development of some some "bomb-proof" failover systems - and it is not great fun as long as it does not work and it might be a lot of work. That is why I would pick up and use existing solutions like clustering databases.

• In order to be fault-tolerant you would need the possibility to run more than one application server in front of the database and some load-balancing mechanism as entry part for such a "JIRA cluster". Such architectures are much easier to implement and maintein if you keep away from all sorts of "caches" except database storage (which is automatically shared by all application servers).
  Sessions are o.k. as long as there are no monstrous objects in them.
  If you use session-sticky load balancers, you can even abandon session propagation along application server cluster nodes.

• We need the indexing architecture. Too long to go into now, but suffice to say that JIRA is somewhere in the order of 100 times more performance using indexes than it is using a database system.

• In terms of re-inventing the wheel, we are using a standard technology called Lucene, which is a standard Java framework. It needs fast R/W access to the indexes. We currently need to store them on disk for this purpose.

We will have a custered version of JIRA, but we need to solve a large number of problems, none of which are trivial (or we would have solved them already). Every now and then I get emails from people who say it should be as easy as flicking a switch in an application server, which probably says more about their experience clustering a large app than it does about how we should go about clustering.

We are aware of most of the patterns involved with custering, but we have not yet had the time to implement them. Also, there is not that much demand for a clustered version, as most people feel that they wouldn't pay 2x the unclustered version to have a clusterable version of JIRA.

If you do find any information on clustering Lucene - please let me know. Currently the best information I have seen is online here:

http://www.mail-archive.com/lucene-user@jakarta.apache.org/msg12709.html

Cheers,
Scott

As I assumed my questions and suggestions were to easy
Perhaps I should not have made a "comment" - if you want, you can delete it.

With your last email you learned me a lot about the JIRA product architecture and its "why-s".

• you are surely right: it will be very complicated to make JIRA cluster-able and significant effort and product redesign will be necessary

• most customers will not cry for it (except customers running enterprise projects as norisbank - in order to get a clusterd system, we should have elected Rational ClearCase and that would have been much, much more expensive)

• I believe "factor 100" for full-text search.
  I used ORACLE and DB2 full-text-search "cartridges" to prevent this problem.
  But as JIRA must be database-independent, you have no such option.

Aktion Winter-Credit bis zum 16.12.2005
Wenn Sie bis zum 16.12.05 einen easyCredit mit Sicherheitsgurt abschließen,
können Sie sich gleich zweimal freuen. Über mehr finanziellen Spielraum und
ein Geschenk von uns: ein schnurloses SIEMENS-Telefon mit vielen Extras.

Den easyCredit mit Sicherheitsgurt erhalten Sie exklusiv in allen norisbank-Filialen,
easyCredit-Shops und in teilnehmenden Volksbanken Raiffeisenbanken.

----Ursprüngliche Nachricht----
Von: Scott Farquhar (JIRA) jira@atlassian.com
Gesendet: Dienstag, 29. November 2005 09:23
An: Rottach, Thilo
Betreff: [JIRA] Commented: (JRA-7330) Implement cluster support

[ http://jira.atlassian.com/browse/JRA-7330?page=comments#action_46918 ]

Scott Farquhar commented on JRA-7330:
-------------------------------------

Thilo,

• We need the indexing architecture. Too long to go into now, but suffice to say that JIRA is somewhere in the order of 100 times more performance using indexes than it is using a database system.

• In terms of re-inventing the wheel, we are using a standard technology called Lucene, which is a standard Java framework. It needs fast R/W access to the indexes. We currently need to store them on disk for this purpose.

We will have a custered version of JIRA, but we need to solve a large number of problems, none of which are trivial (or we would have solved them already). Every now and then I get emails from people who say it should be as easy as flicking a switch in an application server, which probably says more about their experience clustering a large app than it does about how we should go about clustering.

We are aware of most of the patterns involved with custering, but we have not yet had the time to implement them. Also, there is not that much demand for a clustered version, as most people feel that they wouldn't pay 2x the unclustered version to have a clusterable version of JIRA.

If you do find any information on clustering Lucene - please let me know. Currently the best information I have seen is online here:

http://www.mail-archive.com/lucene-user@jakarta.apache.org/msg12709.html

Cheers,
Scott

–
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.atlassian.com/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira

Clustering, generally adds a significant amount of architectual complexity.... here's my two cents on the index issue.

As for managing the index across members of the cluster, if the cluster is member aware (i.e. I'm in the cluster and know that there are two other members as x and y addresses) you could implement a notification over http post, SOAP or RMI. So that if an action on one node invalidates an item in the index, he'll post out to each member of the cluster notifying them that the item should be invalidated. I've seen this method work fairly well. If you're concerned about posting redundantly to all the members, you could use multicast within the cluster group.

Another way to handle this might be to have a service that watches for index changes or invalidations in a member and writes it to the DB and another service that runs on all nodes that polls the DB for index changes or invalidations.

The central index server could work but the added network hop and call out to the index server kind of defeats the purpose.

Do we have plans for it? - Yes
Though there are a few things we need to implement first and I can't really see us having a clustered version with-in a year. We need to prioritise with what our customers work.
For the amount of work required to cluster, we could implement field level security, and a few the other top issues. Only a small percentage of customers actually need and want clustering.
Though we do see it as a necessary feature in the future and would like to get it done done sooner rather than later.

Cheers,
Nick

Sorry.

Cheers,
Nick

For example, both sites create an issue in a project at the same time. They then get the same key.

Clustering is good for failover but not for geographically disperate instance.

Any thoughts on this?

http://terracotta.org/

Terracotta's Session support does not require restricting session or attribute size since it is fine grained. It also does not require any objects inside the graph of sessions (or session attributes) to implement serializable.

Also, Terracotta clusters and makes highly available Lucene RAM indexes (by caching them to disk but keeping arbitrary chunks of the indexes in-heap at any one time).

Should deliver msot of what you need to get JIRA HA with its existing features. Its also open source. Worth a look, because, IMHO JIRA is the best out there at what it does (having used Remedy and Bugzilla) and if you need clustering, Terracotta is probably the fastest path from here to there. Maybe the fine folks at Atlassian would like to work on prebundled integration with us (I work at Terracotta, BTW)?!?

Cheers,

--Ari

Thanks for the suggestion. In the short to medium term we are not planning to look into clustering for JIRA, but concentrate on attacking highly voted for features. We also would like to give Confluence clustering effort to settle down, before we start the work on JIRA.

However, we really appreciate you getting in touch with us!

Cheers,
Anton

http://www.wandisco.com/jiracluster

"JIRA Clustering applies WANdisco's active-active replication technology to bring enterprise class scalability and reliability to JIRA's flexible and easy to use bug tracking, issue tracking, and project management solution. With JIRA Clustering, a central JIRA server is no longer a performance bottleneck.

Features

• Provides clustering over a LAN to dynamically balance workload across multiple JIRA servers at a single location.
• Provides clustering over a WAN to dynamically balance workload across JIRA servers at multiple locations.
• Has no single point of failure whatsoever. JIRA Clustering's approach is truly shared nothing. There is no sharing of disk, CPU, or memory between servers with JIRA Clustering.
• Does not rely on a single shared database. Each server in the cluster has its own independent database replica that is kept in sync with all of the others by WANdisco's active-active replication technology.
• Provides failover and automated disaster recovery over a WAN or a LAN to insure business continuity, without any reliance on third party disk mirroring solutions.
• Can be implemented standalone or in conjunction with WANdisco's clustering solutions for Subversion and CVS to provide a complete, highly reliable and scalable application lifecycle management solution stack.

I'm pleased to announce that Sourcesense (http://www.sourcesense.com) has just launched the first beta of Scarlet, a free, Open Source, Jira clustering extension based on Terracotta DSO (http://terracotta.org).

You can download and learn more about it at : http://confluence.atlassian.com/display/JIRAEXT/Scarlet+-+Terracotta+based+clustering+for+Jira

Feedback welcome!

Sergio B.

http://www.wandisco.com/jiracluster

Enterprise Class MultiSite and Clustering For Market Leading Issue Tracker.

Register url: https://www2.gotomeeting.com/register/847436690

WANdisco, a leading provider of distributed software development solutions, today announced a webinar focused on helping Atlassian JIRA customers leverage their JIRA implementations for scaling and performance. The free, hour-long event, "Scaling JIRA: Because You've Got Lots of Issues, Everywhere!," begins at 11:00am PDT / 2:00pm EDT on August 5, 2008. It is part of WANdisco's LIVE webinar series.

In this webcast, you'll learn how WANdisco's active-active replication, the driver behind JIRA MultiSite and JIRA Clustering, ensures that Atlassian's market leading issue tracker can perform and scale in any environment. This session will not be about industry trends, analyst perspectives, or a deep dive on JIRA features and functions. Instead, we will focus on real world challenges that exist in larger, complex deployments of JIRA, especially those with significant numbers of users or issues.

In addition to a quick overview of how JIRA has come to be used by over 8,700 organizations in more than 97 countries, some of the key topics we'll cover include how to:

• Add failover and automated disaster recovery to JIRA for business continuity, uptime, and to accommodate the demands of global development.

• Employ a solution with no single point of failure from a true shared nothing approach - no sharing of disk, CPU, or memory between servers with JIRA MultiSite or JIRA Clustering.

• Balance workload across multiple servers in any location.

• Add Subversion with replication to create a complete, enterprise ready ALM solution stack.

Space is Limited.

Reserve your seat now at:

https://www2.gotomeeting.com/register/847436690