Subversion import from ClearCase fails near end

Dave Gebhart
Posts: 5
Joined: Wed Apr 12, 2006 7:05 pm

Subversion import from ClearCase fails near end

Postby Dave Gebhart » Wed Apr 12, 2006 7:16 pm

I am importing a moderate size VOB to subversion and it continues to fail towards the end of the process when it takes the CC dump and adds it to SVN. It generates the following error:

java.io.IOException: The pipe has been ended
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(Unknown Source)
at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
at java.io.BufferedOutputStream.write(Unknown Source)
at org.polarion.svnimporter.common.Util.copy(Util.java:303)
at org.polarion.svnimporter.svnprovider.internal.svnadmin.SvnAdminImportExec$1.run(SvnAdminImportExec.java:58)
13:59:58,930 [main] DEBUG Exec:80 - Process exit value: 1
13:59:58,930 [main] INFO historyLogger:84 - EXCEPTION CAUGHT: org.polarion.svnimporter.svnprovider.SvnAdminException: Error during execution command svnadmin.exe load --parent-dir . c:\svn\opensource: process exit code 1
at org.polarion.svnimporter.svnprovider.SvnAdmin.executeCommand(SvnAdmin.java:338)
at org.polarion.svnimporter.svnprovider.SvnAdmin.importDump(SvnAdmin.java:85)
at org.polarion.svnimporter.main.Main.importDump(Main.java:172)
at org.polarion.svnimporter.main.Main.run(Main.java:94)
at org.polarion.svnimporter.main.Main.main(Main.java:49)

13:59:58,930 [main] DEBUG CCProvider:80 - cleanup


I am running version 1.3.0 of svn and version 1.1-M7 of svn-importer. Any help would be much appreciated

Dave

dobisekm
Posts: 118
Joined: Wed Mar 23, 2005 3:29 pm
Location: Prague, Czech Republic

Postby dobisekm » Wed Apr 19, 2006 1:17 pm

Hi Dave,

I don't see the problem cause from the log, but it seems like telling the importer not to import the dump automatically, but just save it to file and then importing the dump file it manually using the 'svnadmin load' command should either solve your problem or at least give some more info about where the problem lies.

I hope this helps,

Michal Dobisek
Polarion Software

Dave Gebhart
Posts: 5
Joined: Wed Apr 12, 2006 7:05 pm

Postby Dave Gebhart » Thu Apr 20, 2006 3:58 pm

I ran the import manually and got the following error:

svnadmin: Found malformed header block in dumpfile stream

The dump file is about 150MB, but it doesn't seem to be adding very many individual files to the repository (less than two dozen files and sub-directories) before the load croaks. Is there a way to validate and/or correct the dump file manually?

Dave

dobisekm
Posts: 118
Joined: Wed Mar 23, 2005 3:29 pm
Location: Prague, Czech Republic

Postby dobisekm » Thu Apr 20, 2006 4:21 pm

Hi Dave,

the dump file is basically the mixture of text headers and binary content, MD5 checksums are used to verify content, but no checksums for the headers (only the lenght is given, so that the headers content doesn't have to be eschaped).

It should be easy to find out in which revision the error occurs (since the svnadmin load outputs the information about which revision is being processed). You can either check it and try to correct manually or maybe examine and point out the error in the dump file.

A possible reason which comes to my mind right now is wrong encoding of some national characters, which causes the properties to be longer than the advertised lenght (since UTF-8 encoded characters take more than 1 byte). We experienced similar error with PVCS import.

I hope this will help you to localize the problem.

Michal Dobisek
Polarion Software

Dave Gebhart
Posts: 5
Joined: Wed Apr 12, 2006 7:05 pm

I keep running into malformed headers

Postby Dave Gebhart » Tue Jun 27, 2006 12:09 am

I have tried 4 different ClearCase repositories and only one has succeeded. The dump file for that VOB was about 15MB. Three others have failed, being about 250-315 MB. Are these files too big? There are no special characters in the files. I am also getting some MD5 checksum errors. Would it help to segment the dump files into smaller units?

dobisekm
Posts: 118
Joined: Wed Mar 23, 2005 3:29 pm
Location: Prague, Czech Republic

Postby dobisekm » Tue Jun 27, 2006 8:19 am

Hi Dave,

several hunderts of MB or some GB of dump file are absolutely no problem - so I don't think the problem is in dump file size.

Are you able to find out in which revision/file the problem occurs and post the problematic sample of the dump file?

Regarding the MD5 checksup errors - we have experienced a similar problem with PVCS import, which sometimes (quite rarely - zeto to 2 times per 2 GB project) did incorrectly checkout the file content and since we check out it twice - once to calculate the MD5 checksum and once to get the actual content - we got the MD5 checksum errors during the import of the dump file - however, this problem was not caused by the importer.

I hope this helps,

Michal Dobisek, Polarion Team

Dave Gebhart
Posts: 5
Joined: Wed Apr 12, 2006 7:05 pm

Postby Dave Gebhart » Tue Jun 27, 2006 8:16 pm

Here is an example of the problem I am encountering:

------- Committed revision 802 >>>

<<< Started new transaction, based on original revision 803
* adding path : trunk/war/icons/target.gif ... done.
* adding path : tags/2006.04.25.iter57/war/icons/target.gif ... done.
* adding path : tags/2006.0.0.rc1/war/icons/target.gif ... done.
* adding path : tags/2006.06.14.iter61/war/icons/target.gif ... done.
* adding path : tags/2006.06.15.iter61/war/icons/target.gif ...svnadmin: Checksum mismatch, file '/tags/2006.06.15.iter61/war/icons/target.gif':
expected: c85ffbc5e6804a988926da0720f76193
actual: 4dded57e9ca27fc54f34c763fa1cb49a

When I dug into the dump file to examine the area that has problems I found that the header describing the tag was there, but there was no content! It just jumped to the next header! I copied the previous content entry and it proceeded along just fine.

Then I got the following error:

------- Committed revision 897 >>>

<<< Started new transaction, based on original revision 898
* adding path : trunk/war/scripts/topNavigation.js ... done.
* adding path : tags/2006.04.25.iter57/war/scripts/topNavigation.js ... done.
* adding path : tags/2006.0.0.rc1/war/scripts/topNavigation.js ... done.
* adding path : tags/2006.06.14.iter61/war/scripts/topNavigation.js ... done.
* adding path : tags/2006.06.15.iter61/war/scripts/topNavigation.js ... done.
* adding path : tags/2006.03.15.iter54/war/scripts/topNavigation.js ... done.
* adding path : tags/2006.0.0/war/scripts/topNavigation.js ... done.
* adding path : tags/2006.06.07.iter60/war/scripts/topNavigation.js ... done.
* adding path : tags/2006.03.29.iter55/war/scripts/topNavigation.js ... done.
svnadmin: Can't remove 'BuildCenterServices\db\transactions\897-1.txn': The directory is not empty.

Any thoughts about this new error?

Dave Gebhart
Posts: 5
Joined: Wed Apr 12, 2006 7:05 pm

Postby Dave Gebhart » Tue Jun 27, 2006 9:49 pm

I cleared out the existing repository and restarted the load, figuring that the filesystem got bogged down due to high levels of activity on my pc. Sure enough, it ran through farther after restarting. Unfortunately, it ran into another instance of missing content (causing a checksum error) and halting the whole process. This is really tiresome. Can the load pick up at the error point if I correct it and retry or do I have to reset everything and start again?

dobisekm
Posts: 118
Joined: Wed Mar 23, 2005 3:29 pm
Location: Prague, Czech Republic

Postby dobisekm » Wed Jun 28, 2006 8:10 am

Hi Dave,

the

Code: Select all

Can't remove 'BuildCenterServices\db\transactions\897-1.txn': The directory is not empty.
error is likely caused by some software on your local PC - e.g. an antivirus checking the folder meanwhile or something like that.

Regarding resuming the import - there is no simple way (e.g. parameter to the import command), but you can edit the dump file to remove the already imported content. However, I am not aware of any software for convenient editation of 1GB files ;-)

Michal


Return to “Polarion SVN Importer (Repository Converter)”

Who is online

Users browsing this forum: No registered users and 6 guests