Wednesday, May 5, 2010

Migrating Unfuddle Tickets to JIRA

I found myself needing to migrate bugs from Unfuddle, which exports them in a custom XML format, to JIRA, which can import CSV (documentation). I threw together a quick Java class to help me do this. It takes backup.xml generated from Unfuddle and creates a CSV which can be read by JIRA. It imports the following fields:
  • Summary
  • Status
  • Description
  • Milestone (as a custom JIRA field)
  • Assignee
  • Reporter
  • Resolution (if resolved)
  • Resolution description (as a comment)
  • Creation time
  • Resolved time (if resolved)
Furthermore it outputs the bugs in the order of the ID in Unfuddle, so that if you're importing into an empty JIRA project, the bugs will have the same number as in Unfuddle. It assumes the JIRA usernames correspond to Unfuddle usernames, though you can easily map differences by modifying the lookupUser function. Once you generate the CSV, you can give the configuration file below to the JIRA CSV Import wizard to take care of the mappings. You'll want to update
  • existingprojectkey
  • user.email.suffix
to match your project. There are a few notable things that are missed with this tool:
  • Time of day for creation/resolved
  • Comments
The tool should run without modification and requires only Joda Time as a dependency under JDK 1.6. This is total slapdash, quick-n-dirty, git-er-done code for a one-off conversion. If anyone would like to extend this tool or generalize it, that would be great :)

Java class UnfuddleToJira.java:

// Original author Gabe Nell. Released under the Apache 2.0 License
// http://www.apache.org/licenses/LICENSE-2.0.html

import java.io.FileOutputStream;
import java.io.PrintStream;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import javax.xml.parsers.DocumentBuilderFactory;

import org.joda.time.DateTime;
import org.joda.time.format.DateTimeFormat;
import org.joda.time.format.DateTimeFormatter;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;

public class UnfuddleToJira {

    private static final DateTimeFormatter DATE_FORMATTER = DateTimeFormat.forPattern("yyyyMMdd");

    private final Document doc;
    private final PrintStream output;
    private final Map<String, String> milestones;
    private final Map<String, String> people;

    public UnfuddleToJira(Document doc, PrintStream output) {
        this.doc = doc;
        this.output = output;
        this.milestones = parseMilestones(doc);
        this.people = parsePeople(doc);
    }

    private static Map<String, String> parseMilestones(Document doc) {
        Map<String, String> milestones = new HashMap<String, String>();
        NodeList milestoneNodes = doc.getElementsByTagName("milestone");
        for (int i = 0; i < milestoneNodes.getLength(); i++) {
            Element elem = (Element)milestoneNodes.item(i);
            String title = elem.getElementsByTagName("title").item(0).getTextContent();
            String id = elem.getElementsByTagName("id").item(0).getTextContent();
            milestones.put(id, title);
        }
        System.out.println("Found " + milestones.size() + " milestones: " + milestones);
        return milestones;
    }

    private static Map<String, String> parsePeople(Document doc) {
        Map<String, String> people = new HashMap<String, String>();
        NodeList peopleNodes = doc.getElementsByTagName("person");
        for (int i = 0; i < peopleNodes.getLength(); i++) {
            Element elem = (Element)peopleNodes.item(i);
            String name = elem.getElementsByTagName("username").item(0).getTextContent();
            String id = elem.getElementsByTagName("id").item(0).getTextContent();
            people.put(id, name);
        }
        System.out.println("Found " + people.size() + " people: " + people);
        return people;
    }

    private static String prepareForCsv(String input) {
        if (input == null) return "";
        return "\"" + input.replace("\"", "\"\"") + "\"";
    }

    private static String convertDate(String input) {
        return DATE_FORMATTER.print(new DateTime(input));
    }

    private String lookupUser(String id) {
        String person = people.get(id);
        /**
         * Here you can transform a person's username if it changed between
         * Unfuddle and JIRA. Eg: <tt> 
         * if ("gabe".equals(person)) {
         *     person = "gabenell";
         * }
         * </tt>
         */
        return person;
    }

    private String lookupMilestone(String id) {
        return milestones.get(id);
    }

    private void writeCsvHeader() {
        StringBuilder builder = new StringBuilder(256);
        builder.append("Summary, ");
        builder.append("Status, ");
        builder.append("Assignee, ");
        builder.append("Reporter,");
        builder.append("Resolution,");
        builder.append("CreateTime,");
        builder.append("ResolveTime,");
        builder.append("Milestone,");
        builder.append("Description");
        output.println(builder.toString());
    }

    private void writeCsvRow(Ticket ticket) {
        StringBuilder builder = new StringBuilder(256);
        builder.append(prepareForCsv(ticket.summary)).append(", ");
        builder.append(prepareForCsv(ticket.status)).append(", ");
        builder.append(prepareForCsv(lookupUser(ticket.assigneeId))).append(", ");
        builder.append(prepareForCsv(lookupUser(ticket.reporterId))).append(", ");
        builder.append(prepareForCsv(ticket.resolution)).append(", ");
        builder.append(prepareForCsv(convertDate(ticket.createdTime))).append(", ");
        String resolveTime = ticket.resolution != null ? convertDate(ticket.lastUpdateTime) : null;
        builder.append(prepareForCsv(resolveTime)).append(", ");
        builder.append(prepareForCsv(lookupMilestone(ticket.milestoneId))).append(", ");
        builder.append(prepareForCsv(ticket.description));

        // JIRA doesn't have the notion of a resolution description, add it as a
        // comment
        if (ticket.resolutionDescription != null) {
            builder.append(",").append(prepareForCsv(ticket.resolutionDescription));
        }
        output.println(builder.toString());
    }

    public void writeCsv() throws Exception {
        NodeList ticketNodes = doc.getElementsByTagName("ticket");
        List<Ticket> tickets = new ArrayList<Ticket>();
        for (int i = 0; i < ticketNodes.getLength(); i++) {
            Node node = ticketNodes.item(i);
            Element nodeElem = (Element)node;
            Ticket ticket = new Ticket();
            NodeList ticketElements = nodeElem.getChildNodes();
            for (int j = 0; j < ticketElements.getLength(); j++) {
                Node ticketSubNode = ticketElements.item(j);
                String nodeName = ticketSubNode.getNodeName();
                if ("id".equals(nodeName)) {
                    ticket.id = ticketSubNode.getTextContent();
                } else if ("status".equals(nodeName)) {
                    ticket.status = ticketSubNode.getTextContent();
                } else if ("summary".equals(nodeName)) {
                    ticket.summary = ticketSubNode.getTextContent();
                } else if ("description".equals(nodeName)) {
                    ticket.description = ticketSubNode.getTextContent();
                } else if ("milestone-id".equals(nodeName)) {
                    ticket.milestoneId = ticketSubNode.getTextContent();
                } else if ("assignee-id".equals(nodeName)) {
                    ticket.assigneeId = ticketSubNode.getTextContent();
                } else if ("reporter-id".equals(nodeName)) {
                    ticket.reporterId = ticketSubNode.getTextContent();
                } else if ("resolution".equals(nodeName)) {
                    ticket.resolution = ticketSubNode.getTextContent();
                } else if ("resolution-description".equals(nodeName)) {
                    ticket.resolutionDescription = ticketSubNode.getTextContent();
                } else if ("created-at".equals(nodeName)) {
                    ticket.createdTime = ticketSubNode.getTextContent();
                } else if ("updated-at".equals(nodeName)) {
                    ticket.lastUpdateTime = ticketSubNode.getTextContent();
                }
            }
            tickets.add(ticket);
        }
        System.out.println("Writing " + tickets.size() + " tickets...");

        // Output to CSV in order of ticket number
        writeCsvHeader();
        Collections.sort(tickets);
        for (Ticket ticket : tickets) {
            writeCsvRow(ticket);
        }
    }

    public static class Ticket implements Comparable<Ticket> {

        public String id;
        public String summary;
        public String status;
        public String description;
        public String milestoneId;
        public String assigneeId;
        public String reporterId;
        public String resolution;
        public String resolutionDescription;
        public String createdTime;
        public String lastUpdateTime;

        @Override
        public int compareTo(Ticket other) {
            return Integer.parseInt(id) - Integer.parseInt(other.id);
        }
    }

    public static void main(String[] args) throws Exception {
        DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
        factory.setNamespaceAware(true);
        if (args.length != 2) {
            System.err.println("Usage: UnfuddleToJira /path/to/unfuddle/backup.xml /path/to/jira/output.csv");
            return;
        }
        String inputFilename = args[0];
        String outputFilename = args[1];
        PrintStream output = new PrintStream(new FileOutputStream(outputFilename), true, "UTF-8");
        UnfuddleToJira converter = new UnfuddleToJira(factory.newDocumentBuilder().parse(inputFilename), output);
        converter.writeCsv();
        output.close();
    }

}

Configuration file:

# written by PropertiesConfiguration
# Wed May 05 07:12:57 UTC 2010
existingprojectkey = WEB
importsingleproject = false
importexistingproject = true
mapfromcsv = false
field.Resolution = resolution
field.Milestone = customfield_Milestone:select
field.Assignee = assignee
field.Summary = summary
field.Status = status
field.Description = description
field.Reporter = reporter
field.CreateTime = created
value.Status.closed = 6
value.Resolution.works_for_me = 5
value.Resolution.will_not_fix = 2
value.Status.new = 1
value.Status.reassigned = 1
value.Resolution.invalid = 4
value.Resolution.postponed = 2
value.Status.accepted = 3
value.Resolution.fixed = 1
value.Resolution.duplicate = 3
user.email.suffix = @kikini.com
date.import.format = yyyyMMdd
field.ResolveTime = resolutiondate
date.fields = CreateTime
date.fields = ResolveTime

14 comments:

  1. I've spent a while trying variations on the commands below, but I always end up with the following error:

    k$ javac -classpath joda-time-1.6.jar UnfuddleToJira.java
    k$ java UnfuddleToJira backup.xml output.csv
    Exception in thread "main" java.lang.NoClassDefFoundError: org/joda/time/ReadableInstant
    Caused by: java.lang.ClassNotFoundException: org.joda.time.ReadableInstant
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)

    But when I do a jar tf joda-time-1.6, ReadableInstant is clearly there. Any idea what I might be doing wrong? Thanks!

    ReplyDelete
  2. Hey Kalvin,

    You're almost there! Just need get both "." and joda-time-1.6.jar on the classpath:

    java -cp joda-time-1.6.jar:. UnfuddleToJira

    Alternately you could extract the jar using

    jar xf joda-time-1.6.jar

    then type the same command you tried.

    Good luck!

    ReplyDelete
  3. OH, that makes total sense. Thanks. So now it runs... but I get this output:

    k$ java -cp joda-time-1.6.jar:. UnfuddleToJira backup.xml output.csv
    Found 13 milestones: {#=name, etc., blah blah}
    Exception in thread "main" java.lang.NullPointerException
    at UnfuddleToJira.parsePeople(UnfuddleToJira.java:56)
    at UnfuddleToJira.(UnfuddleToJira.java:35)
    at UnfuddleToJira.main(UnfuddleToJira.java:202)

    Do you think Unfuddle has changed its XML structure in the last month, or have you seen this issue before? Let me know, thanks!

    ReplyDelete
  4. Hmm, can you take a look at the "person" blocks under "people" in the XML and see if there is a "person" that has an "id' tag but not a "username" tag or with an empty "username" tag? The code is expecting XML with these two fields (your XML may have more, but this is all the code reads):

    <person>
    <id type="integer">36353</id>
    <username>jonv</username>
    </person>

    If the "username" tag is missing or possibly empty we'd need to figure out some way to map the Unfuddle "person" to a JIRA user... either manually sticking it in the XML, or updating the code to check for a missing username and generate one.

    ReplyDelete
  5. The problem turned out to be that deleted users in Unfuddle still show up as person nodes in the XML export, but without their ids-- just their usernames and other info. Removing the deleted users allowed the full import. Thanks so much for your help!

    ReplyDelete
  6. Still getting this:

    java -cp joda-time-1.6.jar:. UnfuddleToJira backup.xml output.csv
    Exception in thread "main" java.lang.NoClassDefFoundError: UnfuddleToJira
    Caused by: java.lang.ClassNotFoundException: UnfuddleToJira
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)

    ReplyDelete
  7. Are you on Windows? Windows uses semicolon as the classpath separator instead of colon.

    java -cp joda-time-1.6.jar;. UnfuddleToJira backup.xml output.csv

    ReplyDelete
  8. No I'm on OSX. Tried semicolon, doesn't work.

    ReplyDelete
  9. I'm on OS X too. Here are the exact step-by-step commands I run, and it works great:


    curl http://static.gabenell.com/blog/201005/UnfuddleToJira.java > UnfuddleToJira.java

    curl http://repo1.maven.org/maven2/joda-time/joda-time/1.6/joda-time-1.6.jar > joda-time-1.6.jar

    javac -classpath joda-time-1.6.jar UnfuddleToJira.java

    java -cp joda-time-1.6.jar:. UnfuddleToJira

    ReplyDelete
  10. OK, got it workin g up to the actual parsing step, getting the following error;

    Konstantin:gabe konstantin$ java -cp joda-time-1.6.jar:. UnfuddleToJira backup.xml output.csv
    Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Arrays.java:3209)
    at java.lang.String.(String.java:215)
    at com.sun.org.apache.xerces.internal.xni.XMLString.toString(XMLString.java:185)
    at com.sun.org.apache.xerces.internal.parsers.AbstractDOMParser.characters(AbstractDOMParser.java:1188)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:464)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
    at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:235)
    at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:284)
    at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:180)
    at UnfuddleToJira.main(UnfuddleToJira.java:202)

    ReplyDelete
  11. Hah, OK, how big is your XML and how much RAM do you have? Like I cautioned, this is a quick-n-dirty tool, and I didn't spend much time to make it efficient :)

    Hopefully you can get it to work by running java with the -Xmx1024M argument, which will give java 1024MB of Heap space. Substitute how a little less than your total RAM for this number to give java as much as possible, Eg:

    java -Xmx1024M -cp joda-time-1.6.jar:. UnfuddleToJira

    ReplyDelete
  12. Thank you so much Gabe, worked like a charm. Now I need to deal with a mess of comments, code snippets, etc. Its a pain dealing with Unfuddle custom stuff.

    ReplyDelete
  13. Here's a version that also imports comments:

    https://gist.github.com/acrollet/11271227

    ReplyDelete
  14. https://github.com/TheLevelUp/unfuddle-jira-migration
    We've written an updated Unfuddle -> JIRA migration script that does a few extras that JIRA's APIs support now, including attachments, epics, custom select field mappings, and issue linking.

    ReplyDelete