Forums

"Ohloh repeatedly fails to download the source"

Hi!

There were many days since the new SVN repo was enlisted in Arch Linux Packages project. I've seen some progress numbers a couple of times, but it always returns to 'Step 1 of 3: Downloading source code history (Waiting in queue)' after some time. Could you please look in logs why it cannot download the code, so if it's a problem on our side we could fix it.

Thank you.

Roman Kyrylych

almost 4 years ago

Hi Roman,

This looks like a problem on our end. The particular server which has been trying to download this code has been having some problems.

I've reassigned the work to a different server. It looks like it's running quickly now. I'll keep an eye on it.

Thanks, Robin

Robin Luckey

almost 4 years ago

Hi Robin, there seem to be the exact same problem on project freecad (hosted on sourceforge)... Last update (failed) is from 10 months ago, I just reset the svn url, but it continues the same... Can you have a look too? Thanks Yorik

Yorik van Havre

almost 4 years ago

@ Robin: thank you for looking into that.

Unfortunately there does not seem to be much progress.

For the past few hours I see the same status message: 'Step 1 of 3: Downloading source code history (Running 0/2)'.

I wonder why it's /2 only, because I do remember it was many thousands during unsuccessful tries to download the source few days before.

Roman Kyrylych

almost 4 years ago

Hi Roman,

There's more going on behind the scenes that I didn't explain.

Our "regular" Subversion importer downloads each revision individually, so there is a progress bar step for each revision. Although this is a robust strategy, the process is very slow.

We have also a "fast" Subversion importer that is based on svnsync. It's much, much faster. However, not all servers work well with this importer, so we only use it by default with Google Code and SourceForge.

Because this is a large repository and because it was giving us trouble, I switched us over to use the svnsync strategy to see if that would work better in this case.

The downside of using svnsync is that it's not trivial to see the progress, so our downloader only shows two steps: one for us to create a local mirror, and another step for us to create a backup of that mirror.

As it turns out, the first step (create a local mirror) has gone well, and we've already downloaded all 45000+ revisions.

We're currently on the second step, creating a backup.

Robin Luckey

almost 4 years ago

Hi yorik,

I'm not sure what the problem may have been with FreeCAD.

By deleting the old enlistment and replacing it with a new one, you've discarded any debugging information we might have had about the old repository.

In any case, things seem to be running normally now -- I don't see any problems. I'll keep an eye on it.

Thanks, Robin

Robin Luckey

almost 4 years ago

@ Robin: thank you for the detailed explanation!

Roman Kyrylych

almost 4 years ago

Thanks Robin, it seems to be working, sorry I deleted the logs! Yorik

Yorik van Havre

almost 4 years ago

@ Robin: the souce was successfully downloaded, but the next step failed:

Step 3 of 3: Counting lines of source code (Failed about 8 hours ago.)

Any ideas?

Roman Kyrylych

almost 4 years ago

I think it failed to count the number of lines again because now I get:

Step 3 of 3: Counting lines of source code (Failed about 4 hours ago.)

Roman Kyrylych

almost 4 years ago

Hi Roman,

I've been watching this job. It looks like we may have some kind of new bug that is causing failures. Give us a few days to investigate.

Thanks, Robin

Robin Luckey

almost 4 years ago

Roman,

This repository was so large that it uncovered a memory leak in our analysis tools.

I've deployed the fix, and the repository now seems to be making progress.

It looks like it will take a few days for the analysis to go through, but I think we are now successfully past the worst of it.

Thanks, Robin

Robin Luckey

almost 4 years ago

Thank you for the support, Robin!

Roman Kyrylych

almost 4 years ago

Hm.. It's been 13 days since Ohloh finished SVN checkout. Since that time it tried to sync from CVS repo (we're in process of fixing it to be accessible again), but never tried to sync from SVN. Is it normal?

Roman Kyrylych

almost 4 years ago

Hi Roman,

As long as any repository is in a failed state, our system won't attempt to update any of the others. One failed repository will block the entire project.

If the CVS is going to down for some time, you might want to remove it from the project temporarily.

Robin Luckey

almost 4 years ago

Okay, thank you.

Roman Kyrylych

almost 4 years ago

Well, I've removed the CVS repository that failed but now the svnsync-based importer fails:

svn://svn.archlinux.org/srv/svn-packages Subversion (via SvnSync) (Failed 7 days ago.)

Though non-svnsync importer seems to be okay, just waiting for the other enlistenment to sync:

svn://aur.archlinux.org/srv/svn-packages Subversion Ohloh update completed 15 days ago.

Roman Kyrylych

over 3 years ago

Hi Roman,

Unfortunately, it looks like this project update is now blocked on a crashing bug in our line counter. This may or may not be a duplicate of another crashing bug that we are already working on. I will open a new ticket, and a fix should be in place soon.

Thanks, Robin